Face-orientation-based attention detection method, electronic device and chip system

By calculating the angle of the face on multiple electronic devices to determine the user's orientation, the problem of users having to switch input device focus between multiple devices is solved, realizing fast and convenient focus switching and improving user experience and interface interactivity.

WO2026232733A1PCT designated stage Publication Date: 2026-11-12HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2025/121534
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-21
Filing Date
2025-09-16
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

When implementing keyboard and mouse sharing across multiple electronic devices, users have to cumbersomely click the mouse or screen to switch the focus of the input device, which reduces the user experience.

Method used

By calculating the angle of the user's face on multiple electronic devices, the system comprehensively determines the user's face orientation and controls the switching of the input device's focus on different electronic devices. By using face orientation judgment, the system can quickly switch the focus of the input device, avoiding tedious clicking operations for the user.

Benefits of technology

It improves the ease of switching input device focus between multiple electronic devices and enhances the user experience, while also increasing the interactivity and user-friendliness of the interface and reducing accidental operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2025121534_12112026_PF_FP_ABST
    Figure CN2025121534_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The embodiments of the present application relate to the technical field of terminals. Provided are a face-orientation-based attention detection method, an electronic device and a chip system. The method comprises: a plurality of electronic devices that share an input device each calculating an included angle between the respective electronic device and the face of a user, so as to comprehensively determine a face orientation of the user, thereby controlling the focus of the input device to switch among different electronic devices. In this way, by means of determining a face orientation, the focus of an input device can be quickly switched among different electronic devices, without requiring a user to cumbersomely switch the focus of the input device by clicking a mouse or tapping a screen, thereby improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Attention detection methods, electronic devices and chip systems based on facial orientation

[0001] This application claims priority to Chinese Patent Application No. 202510592245.6, filed on May 8, 2025, entitled "Attention Detection Method, Electronic Device and Chip System Based on Facial Orientation", and to Chinese Patent Application No. 202510662109.X, filed on May 21, 2025, both of which are incorporated herein by reference in their entirety. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to an attention detection method, electronic device and chip system based on facial orientation. Background Technology

[0003] With the development of the internet age, there may be scenarios where users use multiple electronic devices simultaneously. To facilitate user experience, multiple electronic devices can support keyboard and mouse sharing, meaning that multiple electronic devices can share input devices such as keyboards, mice, or touchpads.

[0004] However, in some scenarios, when implementing keyboard and mouse sharing functionality across multiple electronic devices, users need to cumbersomely click the mouse or screen to switch the focus of the input device, which reduces the user experience. Summary of the Invention

[0005] This application provides an attention detection method, electronic device, storage medium, and chip system based on facial orientation. Multiple electronic devices sharing an input device can each calculate the angle with the user's face, comprehensively determine the user's facial orientation, and thus control the switching of the input device's focus on different electronic devices. In this way, by determining facial orientation, the focus of the input device can be quickly switched between different electronic devices, eliminating the need for the user to cumbersomely switch the focus by clicking a mouse or screen, thereby improving the user experience.

[0006] In a first aspect, embodiments of this application provide an attention detection method based on facial orientation, applied to a first electronic device, the method comprising:

[0007] A first angle between a first electronic device and the user's face is determined, and a second angle from a second electronic device is obtained. The second angle is the angle between the second electronic device and the user's face. The second electronic device responds to input events from the input device. If the first angle is smaller than the second angle, the electronic device responding to the input event switches from the second electronic device to the first electronic device. In response to a user triggering a first operation on the input device, the corresponding input event is executed. It is understood that multiple electronic devices sharing an input device (including the first and second electronic devices) can each calculate their angles with the user's face (including the first and second angles), comprehensively determining the user's face orientation, thereby controlling the switching of the input device's focus across different electronic devices. In this way, by determining the face orientation, the focus of the input device can be quickly switched across different electronic devices, eliminating the need for the user to cumbersomely switch the focus by clicking a mouse or screen, thus improving the user experience.

[0008] In one possible implementation, after the electronic device responding to input events from the second electronic device switches to the first electronic device, the method further includes: displaying a first prompt on the interface of the first electronic device. This first prompt instructs the first electronic device to respond to input events from the input device. It is understood that through this first prompt, such as a mouse cursor tooltip, the user can promptly understand that the focus of the keyboard and mouse has been switched from the second electronic device to the first electronic device. Using a mouse cursor tooltip also helps users quickly locate the mouse pointer, thereby improving the user experience.

[0009] In one possible implementation, the first prompt message includes one or more of the following types: icon-based prompts, text-based prompts, and mouse tooltips. It is understood that icon-based prompts convey information through visual symbols, making it easier for users to identify and understand the meaning of the first prompt message. Text-based prompts help users better understand which electronic device is currently the focus of the input device, making the interface more interactive and user-friendly. Mouse tooltips allow users to quickly locate the mouse pointer or cursor, facilitating further editing or other operations. In other words, through the first prompt message, users can promptly understand that the focus of the keyboard and mouse has been switched from the second electronic device to the first electronic device, thereby improving the user experience of using the hand-eye coordination function.

[0010] In one possible implementation, the mouse tooltip includes an animated mouse pointer displayed on the interface of the first electronic device. This animation includes one or more of the following: changes in mouse pointer size, color, edge glow, and surrounding effects. By using the mouse pointer animation and ripple effects as focus switching indicators, users can quickly identify that the keyboard and mouse focus has shifted to the first electronic device, facilitating rapid mouse pointer location and improving the user experience.

[0011] In one possible implementation, before the electronic device responding to an input event of the input device switches from the second electronic device to the first electronic device, the method further includes: displaying a second prompt on the interface of the first electronic device. The second prompt indicates that by triggering a second operation, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device. Switching the electronic device responding to the input event of the input device from the second electronic device to the first electronic device includes: responding to the second operation, the electronic device responding to the input event of the input device switches from the second electronic device to the first electronic device. In this way, the second prompt helps users quickly understand how to switch the focus of the input device between different electronic devices, improving the accuracy of operation and allowing users to become more proficient in using the keyboard and mouse sharing function and the hand-eye coordination function.

[0012] In one possible implementation, the second operation includes key presses on the input device. It is understood that the first electronic device responds to input events by triggering the second operation. This allows the user to choose whether to switch the focus of the input device according to their needs, increasing the user's selection and settings of certain functions on the electronic device, enhancing interactivity, and thus improving the user experience.

[0013] In one possible implementation, before determining the first angle between the first electronic device and the user's face, and before obtaining the second angle from the second electronic device, the method further includes receiving an operation to set the second electronic device as an electronic device that shares an input device with the first electronic device. It is understood that by setting the second electronic device as an electronic device that shares an input device with the first electronic device, keyboard and mouse sharing and hand-eye coordination can be achieved between the first and second electronic devices. This facilitates users moving the mouse or sharing input devices between multiple electronic devices, resulting in smoother cross-screen operations.

[0014] In one possible implementation, the input device is either included within the first electronic device or independent of it. This can be understood as the input device being included within the first electronic device, or as a built-in device. A built-in device can be tightly integrated with other parts or modules of the first electronic device, eliminating the need for additional input devices or additional setup or connections, offering convenience and speed. An input device independent of the first electronic device can also be understood as an external device. With an external device, the user can choose the type of input device to suit their personal preferences, and the user can also adjust the position and angle of the external device as needed, providing a more flexible input environment.

[0015] In one possible implementation, the input device includes a keyboard or a mouse.

[0016] In one possible implementation, the first electronic device includes a first camera, and the second electronic device includes a second camera. A first angle is the angle between the first camera and the user's face orientation, and a second angle is the angle between the second camera and the user's face orientation. It is understood that by considering the angles associated with multiple electronic devices (including the first and second angles), the user's face orientation can be comprehensively determined. Compared to determining the user's face orientation using a single electronic device, determining the orientation based on the angles of multiple electronic devices can reduce data fluctuations and inaccurate calculation results, minimizing judgment errors and thus more accurately controlling the switching of the input device's focus across different electronic devices.

[0017] In one possible implementation, the method further includes: a first electronic device displaying a first interface; determining a third angle between the first electronic device and the user's face; if the third angle is greater than or equal to a preset angle threshold, adjusting the displayed content of the first interface, or not displaying the first interface. It is understood that adjusting the displayed content of the first interface or not displaying the first interface can make the content displayed on the first interface blurry or obscured, making it impossible for others to clearly identify the content displayed on the first interface, thereby better protecting the user's information.

[0018] In one possible implementation, the method further includes: a first electronic device displaying a second interface; determining a fourth angle between the first electronic device and the user's face; and if the fourth angle is greater than or equal to a preset angle threshold for a duration exceeding a preset time, entering a screen-off state or a screen-locked state. It is understood that, even if the electronic device does not receive any user-triggered operation within the preset time, it can still determine the user's facial orientation. If the user's face is not facing the electronic device within the preset time, it indicates that the user is not viewing the content displayed on the screen, and the electronic device can turn off or lock the screen. If the user's face is facing the electronic device within the preset time, it indicates that the user may be viewing the content displayed on the screen, and the electronic device will not turn off or lock the screen. In this way, the electronic device can more accurately determine the timing for turning off or locking the screen, thereby improving the user experience.

[0019] In one possible implementation, the method further includes: a first electronic device displaying a call interface; determining a fifth angle between the first electronic device and the user's face; and, if the fifth angle is greater than or equal to a preset angle threshold, the first electronic device activating a call mute function. The call mute state may include the electronic device not using audio data, and also the electronic device not sending the collected audio data to the other party's electronic device. This allows for reasonable protection of the user's information based on their head orientation, thereby improving the user experience.

[0020] In one possible implementation, the method further includes: establishing a communication connection between a first electronic device and a third electronic device, with the third electronic device displaying an interface of a first application; if the distance between the first and third electronic devices is less than a preset distance and the sixth angle is less than a preset angle threshold, the first electronic device displays a third prompt message, which includes the icon of the first application; wherein the sixth angle is the angle between the first electronic device and the user's face. It is understandable that the first electronic device can first determine the user's face orientation when displaying the third prompt message. When the first electronic device determines that the user's face is facing it, i.e., when the user's attention shifts from the third electronic device to the first electronic device, the first electronic device then begins displaying the third prompt message. This allows the user to promptly notice the application continuation prompt message, enabling the user to better utilize the application continuation function on the first electronic device, thereby improving the user experience.

[0021] In one possible implementation, before determining the first angle between the first electronic device and the user's face, the method further includes: acquiring an image containing the user's face; determining key point information of the user's face based on the image; determining the user's head pose information based on the key point information, including the yaw angle of the user's head; and determining the first angle between the first electronic device and the user's face, including: determining the first angle based on the user's head pose information and the distance between the camera and the user's face in the horizontal and vertical directions. It is understood that sparse features are a relatively small number of facial key points, which are sufficient to describe the basic structure and features of the face. By calculating sparse features, the computational complexity of the electronic device can be reduced while maintaining an effective description of facial features, thus improving the processing speed and recognition efficiency of the electronic device while maintaining recognition accuracy.

[0022] Secondly, embodiments of this application provide an attention detection method based on facial orientation, applied to a second electronic device, the method comprising:

[0023] In response to a user triggering a third operation on the input device, the system executes the input event corresponding to the third operation; determines a second angle between the second electronic device and the user's face, and sends the second angle to the first electronic device; if the first angle is less than the second angle, the system does not respond to a user triggering a fourth operation on the input device, where the first angle is the angle between the first electronic device and the user's face. It is understood that multiple electronic devices sharing the input device (including the first and second electronic devices) can each calculate the angle with the user's face (including the first and second angles), comprehensively determine the user's face orientation, and thus control the switching of the input device's focus on different electronic devices. In this way, by determining the face orientation, the focus of the input device can be quickly switched on different electronic devices without requiring the user to cumbersomely switch the focus by clicking the mouse or screen, thus improving the user experience.

[0024] Thirdly, embodiments of this application provide an attention detection method based on facial orientation, applied to a communication system, the communication system including a first electronic device and a second electronic device, the method including:

[0025] The second electronic device responds to the input event from the input device; the first electronic device determines a first angle between itself and the user's face, and the second electronic device determines a second angle between itself and the user's face; if the first electronic device determines that the first angle is less than the second angle, the first electronic device responds to the input event from the input device; the first electronic device responds to the user triggering a first operation on the input device by executing the input event corresponding to the first operation. It can be understood that the first electronic device can calculate the first angle, and the second electronic device can calculate the second angle. By comprehensively determining the first and second angles, the user's face orientation can be judged, thereby controlling the switching of the input device's focus on different electronic devices. In this way, by judging the face orientation, the focus of the input device can be quickly switched on different electronic devices, improving the user experience.

[0026] Fourthly, embodiments of this application provide an apparatus for attention detection based on facial orientation. This apparatus may be an electronic device, or a chip or chip system within an electronic device. The apparatus may include a processing unit. The processing unit is used to implement any processing-related method executed by the electronic device in the first aspect or any possible implementation of the first aspect, or in the implementation of the second aspect, or in the implementation of the third aspect.

[0027] When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect, or to implement the method described in the implementation of the second aspect, or to implement the method described in the implementation of the third aspect.

[0028] When the device is a chip or chip system within an electronic device, the processing unit can be a processor. The processing unit executes instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect, or the method described in the implementation of the second aspect, or the method described in the implementation of the third aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0029] Fifthly, embodiments of this application provide an electronic device including one or more processors and a memory, the memory being coupled to one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors being used to invoke the computer instructions to perform the method described in the first aspect or any possible implementation of the first aspect, or to perform the method described in the implementation of the second aspect, or to perform the method described in the implementation of the third aspect.

[0030] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible implementation of the first aspect, or to perform the method described in the implementation of the second aspect, or to perform the method described in the implementation of the third aspect.

[0031] In a seventh aspect, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the method described in the first aspect or any possible implementation of the first aspect, or to perform the method described in the implementation of the second aspect, or to perform the method described in the implementation of the third aspect.

[0032] Eighthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect, or to perform the methods described in the implementation of the second aspect, or to perform the methods described in the implementation of the third aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0033] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0034] It should be understood that aspects four through eight of this application correspond to the technical solutions of aspects one, two, and three of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0035] Figure 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0036] Figure 2 is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0037] Figure 3 is a schematic diagram of an application scenario for an attention detection method provided in an embodiment of this application;

[0038] Figure 4 is a schematic diagram of a mobile phone interface displaying prompt information according to an embodiment of this application;

[0039] Figure 5 is a schematic diagram of an animation effect of displaying a mouse pointer in a mobile phone interface provided in an embodiment of this application;

[0040] Figure 6 is a schematic diagram of a water ripple animation effect around a mouse pointer provided in an embodiment of this application;

[0041] Figure 7 is a schematic diagram of an application settings window in a PC provided in an embodiment of this application;

[0042] Figure 8 is a schematic diagram of a keyboard and mouse sharing interface that displays a device list according to an embodiment of this application;

[0043] Figure 9 is a schematic diagram of a keyboard and mouse sharing interface in a PC provided in an embodiment of this application;

[0044] Figure 10 is a schematic diagram of the interface for setting up an application in a mobile phone according to an embodiment of this application;

[0045] Figure 11 is a schematic diagram of the module interaction of an attention detection method based on face orientation provided in an embodiment of this application;

[0046] Figure 12 is a schematic diagram showing the relationship between the angle between the face orientation and the electronic device provided in an embodiment of this application;

[0047] Figure 13 is a schematic diagram of a method for determining face orientation based on multiple electronic devices according to an embodiment of this application;

[0048] Figure 14 is a schematic diagram of face orientation determination in an application continuity scenario provided by an embodiment of this application;

[0049] Figure 15 is a schematic diagram of a method for protecting screen content based on face orientation according to an embodiment of this application;

[0050] Figure 16 is a schematic diagram of face orientation determination in a call scenario provided by an embodiment of this application;

[0051] Figure 17 is a schematic diagram of an attention detection method based on face orientation provided in an embodiment of this application;

[0052] Figure 18 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0053] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0054] 1. Terminology

[0055] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0056] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0058] With the development of the internet age, users may use multiple electronic devices simultaneously. To improve the user's input experience, multiple electronic devices can support keyboard and mouse sharing, meaning that multiple electronic devices can share input devices such as keyboards, mice, or touchpads.

[0059] Taking keyboard sharing between a personal computer (PC) and a mobile phone as an example, in some scenarios, users can use the keyboard to edit documents and reply to emails on the PC, and also reply to social media messages on the mobile phone. When users switch between the PC and the mobile phone, they need to switch the keyboard input focus between the two devices. For example, when using the PC, the user needs to click the mouse to switch the keyboard input focus to the PC. When using the mobile phone, the user needs to click the mobile phone screen to switch the keyboard input focus to the mobile phone. In other words, when implementing keyboard and mouse sharing functionality between multiple electronic devices, users need to cumbersomely click the mouse or screen to switch the input device focus, which reduces the user experience.

[0060] In view of this, the attention detection method based on face orientation provided in this application allows multiple electronic devices sharing an input device (including the first electronic device and the second electronic device hereinafter) to calculate the angles (including the first angle and the second angle hereinafter) with the user's face, and comprehensively determine the user's face orientation, thereby controlling the switching of the input device's focus on different electronic devices. In this way, by determining the face orientation, the focus of the input device can be quickly switched on different electronic devices without requiring the user to cumbersomely switch the focus by clicking the mouse or screen, thus improving the user experience.

[0061] It is understood that the function proposed in this application to control the focus switching on different electronic devices by determining the orientation of a person's face can also be called a hand-eye coordination function or a hand-eye collaboration function. For ease of description, it will be referred to as a hand-eye coordination function hereafter.

[0062] The electronic device in this application embodiment can also be any form of terminal device, which has keyboard and mouse sharing function and hand-eye coordination function. For example, electronic devices may include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, electronic devices in 5G networks, or electronic devices in future evolved public land mobile networks (PLMNs), etc., and this application embodiment does not limit this.

[0063] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0064] The electronic equipment in this application embodiment may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0065] In this embodiment, the electronic device includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and main memory. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0066] For example, Figure 1 shows a schematic diagram of the hardware structure of an electronic device.

[0067] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may include hardware, software, or a combination of software and hardware.

[0069] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0070] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the aforementioned memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. For example, in the embodiments of this application, the processor 110 can be used to support the facial landmark detection module in performing facial landmark detection, and to support the integrated sensor information platform in performing head pose estimation and face orientation determination, etc.

[0071] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. Furthermore, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121 and / or instructions stored in memory disposed in the processor. For example, in this embodiment, the internal memory 121 may be used to store code related to facial landmark detection, head pose estimation, and user face orientation determination, etc.

[0072] In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1. The cameras 193 can be used to capture still images or videos. For example, in this embodiment, the cameras can be used to capture images containing a user's face.

[0073] Figure 2 is a schematic diagram of the software architecture of an electronic device according to an embodiment of this application.

[0074] For example, the following uses the HarmonyOS system. This example illustrates the software structure of an electronic device. It should be noted that the embodiments of this application can also be applied to other operating systems, such as... system, system, Systems, etc., can also achieve the solution of this application as long as the functions implemented by each functional module are similar to those in the embodiments of this application.

[0075] A layered software architecture can divide software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the HarmonyOS system may include an application layer, a framework layer, a system infrastructure service layer, and a kernel layer, etc.

[0076] The application layer can include system applications and third-party applications, such as keyboard and mouse sharing applications, settings applications, camera applications, gallery applications, video applications, etc. In some scenarios, the application layer can also be called the application layer. In the embodiments of this application, the keyboard and mouse sharing application can instruct the electronic device to display a prompt message indicating screen focus switching. In order to realize the switching of screen focus between multiple electronic devices, the user can turn on the keyboard and mouse sharing function and the hand-eye coordination function in the settings application. The specific way to turn them on can be referred to the relevant descriptions in the embodiments of Figures 7 to 10, which will not be repeated here.

[0077] The framework layer provides multi-language user program frameworks (Java / C / C++ / JavaScript, etc.) for HarmonyOS applications, as well as multi-language framework application programming interfaces for various software and hardware services. In some scenarios, the framework layer can also be called the application framework layer. In this embodiment, the framework layer may include a multimodal sensor data platform (MSDP). The MSDP can be used to estimate the user's head pose and calculate the angle between the user's face and the electronic device, thereby determining the user's face orientation.

[0078] The system foundation service layer is the core capability set of the HarmonyOS system, supporting HarmonyOS in providing application services through the framework layer in multi-device deployment scenarios. The system foundation service layer may include, but is not limited to, one or more of the following components: system basic capability subsystem set, basic software service subsystem set, enhanced software service subsystem set, hardware service subsystem set, proprietary hardware service subsystem set, HarmonyOS driver framework, and hardware abstraction layer.

[0079] The system's basic capabilities subsystem set provides the foundational capabilities for the operation and scheduling of distributed applications across multiple devices within the HarmonyOS system. This set may include, but is not limited to, one or more of the following components: distributed task scheduling, distributed storage and data management, distributed soft bus, Ark runtime, and multi-modal input subsystem.

[0080] For example, in the embodiments of this application, the distributed soft bus provides communication-related capabilities for data transmission and information interconnection between multiple electronic devices, including wireless local area network (WLAN) service capabilities, Bluetooth service capabilities, soft bus, inter-process communication (remote procedure call, RPC) and other communication capabilities.

[0081] A multimodal input subsystem can integrate input from multiple dimensions. For example, based on the kernel subsystem and driver framework, it can receive input events from input devices such as keyboards, mice, touchscreens, and touchpads, and pass these events to applications (e.g., keyboard and mouse sharing applications). Input events can be understood as signals or data generated by input devices, which can be used to represent certain user actions. Input events are the foundation of software system-user interaction. The multimodal input subsystem can achieve user interaction by receiving and processing input events. For example, when a user presses a key on the keyboard, the multimodal input subsystem can receive a keyboard event and execute the corresponding function based on that event.

[0082] The basic software service subsystem provides common and general software services for HarmonyOS, and may consist of subsystems such as graphics and image software services and distributed media software services. The enhanced software service subsystem provides differentiated capability-enhancing software services for different devices, and may consist of subsystems such as tablet business software services and smart screen business software services. The hardware service subsystem provides common and adaptable hardware services for HarmonyOS, and may include subsystems such as power services. The proprietary hardware service subsystem provides differentiated hardware services for different devices, and may include subsystems such as tablet-specific hardware services and wearable-specific hardware services.

[0083] It is understandable that the above service subsystems can be deployed in different device configurations, and each subsystem can be tailored to its functional granularity.

[0084] The HarmonyOS driver framework and hardware abstraction layer form the foundation for the open hardware ecosystem of the HarmonyOS system. They provide hardware capability abstraction at the top and development frameworks and runtime environments for various peripheral drivers at the bottom. For example, in this embodiment, the hardware abstraction layer may include a facial landmark detection module. This module can be used to obtain facial landmark information based on images captured by a camera.

[0085] HarmonyOS employs a multi-kernel design, which may include, for example, the Linux kernel, the Harmony microkernel, and LiteOS. This design allows electronic devices with varying capabilities to select the appropriate system kernel.

[0086] The methods of this application will be described in detail below through specific embodiments. The following embodiments can be combined with each other or implemented independently. For the same or similar concepts or processes that may exist in some embodiments, they will not be described again.

[0087] For example, the following uses a PC and a mobile phone as examples to illustrate a usage scenario of the attention detection method of this application embodiment. It is understood that the method of this application embodiment can implement attention detection between two or more electronic devices. The specific number and type of electronic devices involved are not limited in this application embodiment.

[0088] In possible scenarios, users can view or edit documents on a PC and chat on a mobile phone. As shown in Figure 3a, the PC interface 300 may include a document interface 301. The specific content displayed on the document interface 301 is not limited in this embodiment.

[0089] A chat window can be displayed on the phone's interface 310. This chat window may include a title bar 311, a chat content display area 312, and a chat content input area 313. The title bar 311 may include a back control and a friend's name. The back control can be used to exit the current chat window. The chat content display area 312 can be used to display the user's chat history with friends. The specific content of the chat history is not limited in this embodiment. The chat content input area 313 can be used to input voice, text, emoticons, or send pictures, videos, etc. The chat content input area 313 may include an input bar 314, which can be used to input chat content such as voice, text, and emoticons.

[0090] It should be noted that the interfaces provided in the embodiments of this application are all exemplary interfaces. Each interface may include more or fewer content and / or controls, and the specific content and / or controls included in each interface, as well as the display style of the content and / or controls, are not limited in the embodiments of this application.

[0091] When the user enables the keyboard and mouse sharing function and the hand-eye coordination function on both the PC and the mobile phone, the PC and the mobile phone can share the keyboard 320 and / or mouse 330. The specific methods for enabling the keyboard and mouse sharing function and the hand-eye coordination function on the PC and the mobile phone can be referred to the relevant descriptions of the embodiments corresponding to Figures 7 to 10 below, and will not be repeated here. The PC and the mobile phone can control which electronic device the focus of the keyboard 320 and / or mouse 330 is on by determining the user's facial orientation. Wireless connections can be established between the PC and the keyboard 320, and between the PC and the mouse 330, and between the mobile phone and the keyboard 320, and between the mobile phone and the mouse 330. Wireless connections include, for example, Bluetooth connections and Wi-Fi connections.

[0092] When a user edits or views document content on the PC's document interface 301, the user can input the content they want to edit using the keyboard 320, and can also view the content on the document interface 301 by sliding or dragging the mouse 330. At this time, the focus of both the keyboard 320 and the mouse 330 is on the PC's interface 300. A cursor can be displayed on the document interface 301, which indicates the position of the mouse pointer or the location of the input content. The user can input the content they want to edit at the cursor position using the keyboard 320.

[0093] When the user's head turns from the PC to the mobile phone, the focus of the keyboard 320 and the mouse 330 will switch to the mobile phone's interface 310 without the user needing to drag or click the mouse 330. The user can enter chat content in the input field 314 using the keyboard 320, and can also view the chat history in the chat content display area 312 by sliding or dragging the mouse 330. At this time, a cursor can be displayed in the input field 314, which is used to indicate the position of the mouse pointer or the position of the input content. The user can enter chat content at the cursor position using the keyboard 320.

[0094] When the user turns their head from the phone back to the PC, without the user needing to drag or click the mouse 330, the focus of the keyboard 320 and the mouse 330 will switch to the PC's interface 300, and the cursor can be displayed in the document interface 301. The user can then use the keyboard 320 to input the content they want to edit in the document interface 301, and can also view the content in the document interface 301 by sliding or dragging the mouse 330.

[0095] In other words, without the need for frequent dragging or clicking of the mouse 330, this embodiment of the application can control the switching of the focus of the keyboard 320 and the mouse 330 between the PC interface 300 and the mobile phone interface 310 by determining the user's facial orientation. This eliminates the need for the user to switch focus between different electronic devices, thus improving the user experience.

[0096] As shown in Figure 3b, a camera 302 is installed on the PC, and a camera 315 is installed on the mobile phone. When the user's head turns from the PC to the mobile phone, the camera 302 can capture an image containing the user's face. The PC can then calculate the angle between the straight line between the camera 302 and the user's face and the direction the user's face is facing (hereinafter referred to as the first angle) based on the image captured by the camera 302. The camera 315 can also capture an image containing the user's head. The mobile phone can then calculate the angle between the straight line between the camera 315 and the user's face and the direction the user's face is facing (hereinafter referred to as the second angle) based on the image captured by the camera 315.

[0097] After obtaining the first and second included angles, the PC or mobile phone can determine the user's face orientation by comparing the first and second included angles. Taking the PC as an example to determine the first and second included angles, the mobile phone can send the calculated second included angle to the PC. Then, the PC can determine the user's face orientation by comparing the sizes of the first and second included angles.

[0098] As shown in Figure 3b, the first angle is greater than the second angle, indicating that the angle between the user's face and the PC is relatively large, while the angle between the user's face and the phone is relatively small. This can also be interpreted as the user's face being turned towards the phone, and the user's attention being focused on the phone. Therefore, the focus of the keyboard 320 and the mouse 330 can be switched from the PC interface 300 to the phone interface 310.

[0099] Optionally, when the user's face is facing the phone, a prompt message can be displayed on the phone's interface 310. This prompt message is used to inform the user that they are currently looking at the phone, and can also be used to instruct the user to switch focus. Optionally, the prompt message can be presented in the form of an icon and / or a text message. The prompt message can be displayed in any possible location on the interface 310; this embodiment does not limit the display location of the prompt message.

[0100] For example, Figure 4 shows a schematic diagram of a prompt message 401 displayed on the mobile phone interface 400. This prompt message is used to inform the user that they are currently looking at the mobile phone, and can also be used to instruct the user to switch focus. The prompt message 401 may include an icon and a prompt. The icon can be a smiley face icon or other icons, which is not limited in this embodiment. The prompt message may include "Press the Ctrl key to switch the cursor to this screen," and may include more or less information, which is not limited in this embodiment. When the user presses the Ctrl key, in response to the user's Ctrl key activation, the mobile phone interface 400 may display a prompt message indicating a focus switch. This focus switch prompt message may be presented in the form of an icon, a prompt message, and / or a mouse pointer.

[0101] Optionally, users can also switch the cursor by triggering other keys (such as the Enter key, Shift key, or space bar) or other operations (such as clicking the mouse). It is understood that the operations mentioned in the embodiments of this application may include, but are not limited to, touch (e.g., clicking), voice control, gestures, etc., and the embodiments of this application do not impose any limitations on them.

[0102] For example, taking the display of the mouse pointer as a prompt message indicating a focus switch, in response to the user pressing the Ctrl key, the mobile phone can display the mouse pointer using the animation effect shown in Figure 5.

[0103] As shown in Figure 5a, a mouse pointer 501 can be displayed on the mobile phone's interface 500. Optionally, the mouse pointer 501 can be displayed at any possible location on the interface 500. This embodiment of the application does not limit this. For example, it can be displayed at the midpoint of the mobile phone's interface 500, the input bar 314 or its vicinity, or the last position where the mouse pointer 501 appeared when the focus was on the mobile phone's interface 500. When displaying the animation effect of the mouse pointer 501, the mouse pointer 501 can gradually enlarge. For example, the enlargement process of the mouse pointer 501 can include a change from Figure 5a to Figure 5b, and then a change from Figure 5b to Figure 5c. It is understood that the interface shown in Figure 5 can be one of the interfaces during the gradual enlargement process of the mouse pointer 501.

[0104] After the mouse pointer 501 is magnified to a certain extent, it can be gradually reduced in size. For example, the reduction process of the mouse pointer 501 can include changing from c in Figure 5 to b in Figure 5, and then changing from b in Figure 5 to a in Figure 5. Optionally, the reduced mouse pointer 501 can be the same size as the mouse pointer 501 in a in Figure 5, or it can be different in size; this embodiment of the application does not limit this.

[0105] Optionally, the style of the mouse pointer 501 can also change during the display of the animation effect. This style change may include a change in the color of the mouse pointer 501 and / or a change in the glowing effect around the mouse pointer 501. Color changes may include changes in hue, brightness, and / or transparency, etc., which are not limited in this embodiment.

[0106] Taking the style changes of mouse pointer 501, including changes in color brightness and glowing effects, as an example, in a possible implementation, as mouse pointer 501 gradually enlarges, its color brightness can become brighter, and the glowing effect around it can become more pronounced. As mouse pointer 501 gradually shrinks, its color brightness can become dimmer, and the glowing effect around it can become less pronounced, or even disappear entirely.

[0107] Optionally, during the display of the animation effect of the mouse pointer 501, other animation effects may also be displayed around the mouse pointer 501. The specific display style of these other animation effects is not limited in this embodiment.

[0108] For example, taking a water ripple animation effect as an example, Figure 6 shows a schematic diagram of a water ripple animation effect displayed around the mouse pointer. As shown in Figure 6a, the mouse pointer 601 can be displayed on the phone's interface 600. Optionally, the mouse pointer 601 can be displayed at any possible position on the interface 600. This embodiment of the application does not limit this. For example, it can be displayed at the midpoint of the phone's interface 600, the input bar 314 or its vicinity, or at the last position where the mouse pointer 601 last appeared when the focus was on the phone's interface 600. During the animation effect of displaying the mouse pointer 601, a water ripple animation effect can be displayed around the mouse pointer. As shown in Figures 6b and 6c, as the mouse pointer 601 gradually enlarges, water ripples 602 can be displayed around the mouse pointer 601. Furthermore, the water ripples 602 can gradually increase in size as the mouse pointer 601 increases, and become increasingly obvious.

[0109] After the mouse pointer 601 is magnified to a certain extent, it will gradually shrink. During this shrinking process, as shown in Figure 6d, water ripples 602 are still displayed around the mouse pointer 601. Furthermore, the water ripples 602 continue to increase in size and become increasingly less noticeable. This display method better simulates the changing process of water ripples, making the animation effect more realistic. It can be understood that the interfaces displayed in Figures 6b, 6c, and 6d can be specific interfaces within the water ripple animation effect display process.

[0110] Once the animation effect of the mouse pointer 601 is complete, the animation effect of the water ripples 602 will also be completed. As shown in Figure 6e, the interface 600 will display the mouse pointer 601, but the water ripples 602 will no longer be displayed.

[0111] In this way, by using the change of the mouse pointer and the animation effect of water ripples as prompts for focus switching, users can know in a timely manner that the focus of the keyboard and mouse has been switched to the mobile phone, making it easier for users to quickly find the position of the mouse pointer and improving the user experience.

[0112] The following sections, referring to Figures 7 through 10, explain how to enable the keyboard and mouse sharing function and the hand-eye coordination function on a PC and a mobile phone, respectively. Figures 7 through 9 show how to enable these functions on a PC, while Figure 10 shows how to enable them on a mobile phone. It is understood that enabling these functions on other electronic devices is similar to that on a PC or mobile phone, and will not be elaborated further.

[0113] For example, as shown in Figure 7, this is a schematic diagram of the interface of a settings application window 701 in a PC interface. Window 701 may include a first display area 702 and a second display area 703. The first display area 702 is used to display the icon of the settings application, the settings title, the search box, account information, and multiple settings items, etc. The second display area 703 is used to display the detailed interface of a certain settings item.

[0114] In the first display area 702, a search box can be used to quickly search for content the user wants to find. The search box may include a search icon and search suggestions, whereby the search suggestions indicate to the user that they can quickly find the content they want by entering keywords. In response to the user triggering the search icon, the PC can search for relevant content based on the keywords in the search box.

[0115] Account information may include the user's profile picture, username, and account identifier (ID).

[0116] Multiple settings can include, for example, WLAN, Bluetooth, network, multi-device collaboration, desktop and personalization, display and brightness, sound, notifications and status bar, system settings, etc. Each setting can include a setting icon and a setting name. For example, taking multi-device collaboration setting 704 as an example, multi-device collaboration setting 704 can include a multi-device collaboration icon 7041 and a multi-device collaboration name 7042.

[0117] In response to a user triggering the multi-device collaboration settings 704, the PC can display a keyboard and mouse sharing interface 705 in the second display area 703. This triggering operation can include triggering any location within the multi-device collaboration settings 704.

[0118] Optionally, in response to a user triggering an action on multi-device collaboration setting 704, multi-device collaboration setting 704 may be displayed differently from other settings. For example, the background (such as color) of multi-device collaboration setting 704 may be different from other settings, or the multi-device collaboration icon 7041 and / or the multi-device collaboration name 7042 may be highlighted.

[0119] It is understood that, in the embodiments of this application, the highlighting display may include, but is not limited to, one or more of the following display methods: changing the color of the font and / or icon, changing the size of the font and / or icon, changing the grayscale of the font and / or icon, and making the font bold.

[0120] In the second display area 703, the keyboard and mouse sharing interface 705 may include a return control, a keyboard and mouse sharing title, a full-screen control, a minimize control, a close control, a keyboard and mouse sharing icon, a keyboard and mouse sharing prompt, a keyboard and mouse sharing option 706, a prompt message corresponding to option 706, a keyboard and mouse sharing device option 707, an option to allow the keyboard and mouse to move to the screen of a nearby device to connect to the device 708, a prompt message corresponding to option 708, a hand-eye coordination option 709, and a prompt message corresponding to option 709, etc.

[0121] The back control is used to exit the current keyboard and mouse sharing interface or return to the previous interface. The maximize control is used to display the settings application window 701 in full screen. The minimize control is used to hide or minimize the display window 701, such as shrinking the window 701 to the PC's taskbar so that the window 701 does not occupy the PC's screen space. The close control is used to close the window 701.

[0122] Keyboard and mouse sharing prompts may include, for example, "Move your mouse to the edge of the screen on your computer, and then continue moving it to the screen of another nearby device to experience cross-device keyboard and mouse sharing and file dragging. Learn more." Keyboard and mouse sharing prompts may also include more or less information, which is not limited in this embodiment. The "Learn more" prompt may be displayed as a hyperlink. In response to a user triggering the "Learn more" action, the PC can open and display the details within the hyperlink. Optionally, "Learn more" may be highlighted, allowing users to quickly locate the hyperlink and easily view its details, thereby improving the user experience.

[0123] The keyboard and mouse sharing option 706 may include an option name (e.g., "Keyboard and Mouse Sharing") and a switch control 7061 for the keyboard and mouse sharing function. The switch control 7061 can be used to turn the keyboard and mouse sharing function on (or enable / disable). For example, when the switch control 7061 is in the off state, the keyboard and mouse sharing function is disabled. In response to a user triggering the switch control 7061, the switch control 7061 may be displayed as on (as shown in Figure 7), and the PC can enable the keyboard and mouse sharing function. When the switch control 7061 is in the on state, the keyboard and mouse sharing function is enabled. In response to a user triggering the switch control 7061, the switch control 7061 may be displayed as off, and the PC can disable the keyboard and mouse sharing function.

[0124] The prompt message corresponding to option 706 may include, for example, "This service requires the use of Bluetooth or WLAN for sharing mouse and keyboard operations between devices. By turning on the switch, you agree to the above." The prompt message corresponding to option 706 may also include more or less information, which is not limited in this embodiment.

[0125] The device option 707 for keyboard and mouse sharing may include the option name of option 707 (e.g., "devices for keyboard and mouse sharing"), a prompt message 7071, and a drop-down control 7072.

[0126] The prompt message 7071 is used to indicate the connection status of the device currently sharing the keyboard and mouse with the PC. For example, if the PC is not currently connected to any keyboard and mouse sharing device, prompt message 7071 can display "No device connected". If the PC is currently connected to a keyboard and mouse sharing device, prompt message 7071 can display relevant information about the sharing device. It is understood that the device's relevant information can be used to uniquely identify the device, making it easier for users to distinguish the connected device. For example, the device's relevant information may include, but is not limited to, one or more of the following: device model, device icon, device name, device ID, and the username of the user logging into the device. The specific content displayed in prompt message 7071 is not limited in this embodiment, as long as the content displayed in prompt message 7071 corresponds to the information of the keyboard and mouse sharing device connected to the PC.

[0127] The dropdown control 7072 can be used to display a list of devices that can be shared with the PC via keyboard and mouse.

[0128] Option 708, which allows the keyboard and mouse to move within the screen of nearby devices to connect, may include an option name (e.g., "Allow keyboard and mouse to move within the screen of nearby devices to connect") and a switch control 7081. The switch control 7081 can be used to turn the function of allowing the keyboard and mouse to move within the screen of nearby devices to connect on or off. For example, when the switch control 7081 is in the off state, the PC does not allow the keyboard and mouse to move within the screen of nearby devices to connect. In response to a user triggering an operation on the switch control 7081, the switch control 7081 may be displayed in the on state (as shown in Figure 7), and the PC allows the keyboard and mouse to move within the screen of nearby devices to connect. When the switch control 7081 is in the on state, the PC allows the keyboard and mouse to move within the screen of nearby devices to connect. In response to a user triggering an operation on the switch control 7081, the switch control 7081 may be displayed in the off state, and the PC does not allow the keyboard and mouse to move within the screen of nearby devices to connect.

[0129] The prompt message corresponding to option 708 may include, for example, "When the mouse moves to the edge of the screen, it can continuously move to the screen of another nearby device logged in with the same account to connect to the device." The prompt message corresponding to option 708 may also include more or less information, which is not limited in this embodiment.

[0130] The hand-eye coordination option 709 may include an option name (e.g., "Hand-eye Coordination") and a switch control 7091 for the hand-eye coordination function. The switch control 7091 can be used to turn the hand-eye coordination function on or off. For example, when the switch control 7091 is in the off state, the hand-eye coordination function is disabled. In response to a user triggering the switch control 7091, the switch control 7091 may be displayed in the on state (as shown in Figure 7), and the PC can enable the hand-eye coordination function. When the switch control 7091 is in the on state, the hand-eye coordination function is enabled. In response to a user triggering the switch control 7091, the switch control 7091 may be displayed in the off state, and the PC can disable the hand-eye coordination function.

[0131] The prompt message corresponding to option 709 may include, for example, "When sharing keyboard and mouse, you can switch the cursor to the screen you are looking at by looking at the screen and using the Ctrl key. This function needs to be enabled on both devices. How to use it?". The prompt message corresponding to option 709 may also include more or less information, which is not limited in this embodiment. The "How to use it?" in the prompt message corresponding to option 709 may be displayed as a hyperlink. In response to the user triggering the "How to use it?" operation, the PC can open and display the details in the hyperlink corresponding to "How to use it?". Optionally, "How to use it?" may be highlighted, so that the user can quickly find the hyperlink and easily view the details, thereby improving the user experience.

[0132] In response to a user triggering the operation of the drop-down control 7072 in the device option 707 for keyboard and mouse sharing, the PC can display a device list that can be used to display options for one or more devices that can be shared with the PC for keyboard and mouse.

[0133] It is understood that, since one or more keyboard and mouse sharing devices are selected on the PC, the PC can act as the master device, and the other keyboard and mouse sharing devices can act as slave or secondary devices, respectively. The PC determines the user's face orientation by judging the included angles reported by the multiple slave devices. Optionally, the master device can also be any electronic device among the multiple keyboard and mouse sharing devices; this embodiment of the application does not limit this. For ease of description, the PC will be used as the master device in the following description.

[0134] For example, as shown in Figure 8a, the PC can display a device list 803 near the drop-down control 802 in the keyboard and mouse sharing interface 801. Optionally, the device list 803 can be displayed in any possible location in the keyboard and mouse sharing interface 801, and this embodiment of the application is not limited thereto. Taking a tablet and a mobile phone as devices that can share a keyboard and mouse with a PC as examples, the device list 803 can display a tablet option 8031 ​​and a mobile phone option 8032. The tablet option 8031 ​​can include an icon and an account name corresponding to the tablet, for example, the tablet's account name is "My Tablet". The mobile phone option 8032 can include an icon and an account name corresponding to the mobile phone, for example, the mobile phone's account name is "My Mobile Phone".

[0135] Users can select one or more devices from device list 803 that they wish to share keyboard and mouse with the PC. For example, in response to the user triggering the operation of tablet option 8031, as shown in Figure 8b, a check icon 8033 can be displayed in tablet option 8031. Then the PC and tablet can establish a communication connection to share keyboard and mouse.

[0136] Optionally, in response to a user triggering an action on tablet option 8031, tablet option 8031 ​​can be displayed differently from other device options. For example, the background (such as color) of tablet option 8031 ​​can be different from other settings, or the icons and / or text in tablet option 8031 ​​can be highlighted.

[0137] Similarly, in response to the user triggering the operation of mobile option 8032, as shown in Figure 8b, a checkmark icon 8034 can be displayed in mobile option 8032. Then, the PC and mobile phone can establish a communication connection for keyboard and mouse sharing.

[0138] After the user selects one or more devices to share the keyboard and mouse with the PC, in response to the user's operation in an area outside the device list 803 of the keyboard and mouse sharing interface 801, the PC can close (or, in other words, not display) the device list 803 and display the keyboard and mouse sharing interface 901 as shown in Figure 9a. Alternatively, if all device options in the device list 803 are selected, the PC can close the device list 803 and display the keyboard and mouse sharing interface 901 as shown in Figure 9a.

[0139] As shown in Figure 9a, in the keyboard and mouse sharing interface 901, the prompt information 9021 of the keyboard and mouse sharing device option 902 no longer displays the "No device connected" prompt information as shown in Figure 7, but instead displays the relevant information of the connected keyboard and mouse sharing device. The specific device information can be found in the device information involved in the prompt information 7071 in the corresponding embodiment of Figure 7, and will not be repeated here. Taking the account name of the logged-in device as an example, the prompt information 7071 may include "My Tablet" and "My Phone". The specific device information displayed in the prompt information 7071 is not limited in this embodiment, as long as it can uniquely identify the device and facilitate user differentiation of the connected device.

[0140] When a communication connection is established between the PC and the keyboard and mouse sharing device, the keyboard and mouse sharing interface 901 can also display screen arrangement options 903. Screen arrangement options 903 may include an option name (e.g., "Screen Arrangement") and an editing control 9031. In response to the user triggering the editing control 9031, as shown in Figure 9b, the PC can display a screen arrangement pop-up window 904 in the keyboard and mouse sharing interface 901. The screen arrangement pop-up window 904 may include a screen arrangement title, a screen arrangement area 9041, and a completion control 9042.

[0141] Within the screen arrangement area 9041, users can adjust the relative positions and screen arrangement order between the PC and other devices sharing the keyboard and mouse. In response to the user triggering the completion control 9042, the PC can display the keyboard and mouse sharing interface 901 as shown in Figure 9a, and the PC can implement the keyboard and mouse sharing function according to the adjusted screen arrangement order. For example, if the user sets the PC screen to the left side of the tablet, when the user moves the mouse to the right edge of the PC screen, the mouse cursor will automatically appear on the tablet screen. Thus, based on the screen arrangement order, users can naturally move the mouse between electronic devices, resulting in smoother cross-screen operations.

[0142] Figure 10 shows how to enable keyboard and mouse sharing and hand-eye coordination on a mobile phone.

[0143] As shown in Figure 10a, this is the interface 1000 of the settings application on the mobile phone. The interface 1000 may include a search bar, an account area, and multiple settings items.

[0144] The search bar instructs the phone to quickly search for user-entered content. It can include a search icon and search suggestions. The search suggestions indicate that users can quickly find the desired content by entering keywords. In response to the user triggering the search icon, the phone can search for relevant content based on the keywords entered in the search box.

[0145] The account area can display account information, account avatar, cloud storage information, etc. It can also include cloud storage controls and a device finder control. Responding to user actions with the cloud storage controls, users can store and manage their data in the cloud, including pictures, videos, contacts, memos, etc., and keep data synchronized across multiple electronic devices logged into the user's account. Responding to user actions with the device finder control, users can quickly locate their mobile phone.

[0146] Multiple settings can include WLAN, Bluetooth, mobile network, satellite network, multi-device collaboration, desktop and personalization, display and brightness, sound and vibration, notification and status bar, general settings, etc. Each setting can include a setting icon, a setting name, and a control to access the setting details page. For example, taking the multi-device collaboration setting as an example, the multi-device collaboration setting can include a multi-device collaboration icon 1001, a multi-device collaboration name 1002, and a control 1003 to access the multi-device collaboration interface.

[0147] In response to a user's triggering of an operation to enter the multi-device collaboration interface, as shown in Figure 10b, the mobile phone can display the multi-device collaboration interface 1004. This triggering operation can include triggering any location within the multi-device collaboration settings, such as triggering the control 1003 to access the multi-device collaboration details page. The multi-device collaboration interface 1004 can include multiple settings, such as sharing, connection, cross-device clipboard, cross-device interoperability, and keyboard / mouse sharing. Each setting can include a setting name and a control to access the setting details page. For example, taking the keyboard / mouse sharing setting as an example, the keyboard / mouse sharing setting can include a keyboard / mouse sharing name 1005 and a control 1006 to access the keyboard / mouse sharing interface.

[0148] In response to a user's triggering of an operation to enter the keyboard and mouse sharing interface, as shown in Figure 10c, the mobile phone can display the keyboard and mouse sharing interface 1007. This triggering operation can include triggering any location within the keyboard and mouse sharing settings, such as triggering control 1006. The keyboard and mouse sharing interface 1007 can include a return control, a keyboard and mouse sharing title, a keyboard and mouse sharing icon, a keyboard and mouse sharing prompt, a keyboard and mouse sharing option 1008, and corresponding prompt information for option 1008.

[0149] The return control is used to exit the current keyboard and mouse sharing interface or return to the previous interface.

[0150] Keyboard and mouse sharing prompts may include, for example, "By dragging the mouse to the edge of the screen and across to another device's screen, you can share your keyboard and mouse across devices and drag and drop files." Keyboard and mouse sharing prompts may also include more or less information, which is not limited in this embodiment.

[0151] The keyboard and mouse sharing option 1008 may include an option name (e.g., "Keyboard and Mouse Sharing") and a switch control 1009 for the keyboard and mouse sharing function. The switch control 1009 can be used to turn the keyboard and mouse sharing function on or off. For example, when the switch control 1009 is in the off state (as shown in Figure 10c), the keyboard and mouse sharing function is disabled. In response to a user triggering the switch control 1009, the switch control 1009 may be displayed in the on state (as shown in Figure 10d), and the phone can enable the keyboard and mouse sharing function. When the switch control 1009 is in the on state, the keyboard and mouse sharing function is enabled. In response to a user triggering the switch control 1009, the switch control 1009 may be displayed in the off state, and the phone can disable the keyboard and mouse sharing function.

[0152] The prompt message corresponding to option 1008 may include, for example, "This service requires the use of Bluetooth or WLAN for sharing mouse and keyboard operations between devices. By turning on the switch, you agree to the above." The prompt message corresponding to option 1008 may also include more or less information, which is not limited in this embodiment.

[0153] In response to the user's operation of triggering the switch control 1009, the mobile phone can display the keyboard and mouse sharing interface 1007 as shown in Figure 10d. Compared with the keyboard and mouse sharing interface 1007 shown in Figure 10c, the switch control 1009 in the keyboard and mouse sharing interface 1007 shown in Figure 10d is displayed in the open state, and the keyboard and mouse sharing interface 1007 may also include the hand-eye coordination option 1010.

[0154] The hand-eye coordination option 1010 may include an option name (e.g., "Hand-eye Coordination") and a switch control 1011 for the hand-eye coordination function. The switch control 1011 can be used to turn the hand-eye coordination function on or off. For example, when the switch control 1011 is in the off state, the hand-eye coordination function is disabled. In response to a user triggering the switch control 1011, the switch control 1011 may be displayed in the on state (as shown in d of Figure 10), and the phone can enable the hand-eye coordination function. When the switch control 1011 is in the on state, the hand-eye coordination function is enabled. In response to a user triggering the switch control 1011, the switch control 1011 may be displayed in the off state, and the phone can disable the hand-eye coordination function.

[0155] The prompt message corresponding to option 1010 may include, for example, "When sharing keyboard and mouse, you can switch the cursor to the screen you are looking at by looking at the screen and using the Ctrl key. This function needs to be enabled on both devices. How to use it?". The prompt message corresponding to option 1010 may also include more or less information, which is not limited in this embodiment. The "How to use it?" in the prompt message corresponding to option 1010 may be displayed as a hyperlink. In response to the user triggering the "How to use it?" operation, the mobile phone can open and display the details in the hyperlink corresponding to "How to use it?". Optionally, "How to use it?" may be highlighted, so that the user can quickly find the hyperlink and easily view the details, thereby improving the user experience.

[0156] Optionally, in Figure 10b, in response to a user triggering an operation to enter the keyboard and mouse sharing interface (e.g., triggering the operation of control 1006), the mobile phone can also display the keyboard and mouse sharing interface 1007 as shown in Figure 10d, instead of displaying the keyboard and mouse sharing interface 1007 as shown in Figure 10c. That is, when displaying the keyboard and mouse sharing interface, the interface can include both the keyboard and mouse sharing option and the hand-eye coordination option simultaneously, instead of only displaying the hand-eye coordination option when the keyboard and mouse sharing function is enabled.

[0157] The following, with reference to the software architecture in Figure 2, uses a PC and a mobile phone as examples to illustrate the module interaction diagram of the attention detection method based on face orientation in this application embodiment. It is understood that both the PC and the mobile phone can include various functional modules or applications implementing the method of this application embodiment, such as a face key point detection module, a comprehensive sensor information platform, a multi-modal input subsystem, and a keyboard and mouse sharing application, etc.

[0158] For ease of description and distinction, a PC can be referred to as the first electronic device and a mobile phone as the second electronic device. The PC may include a first camera, a first facial landmark detection module, a first integrated sensor information platform, a first multi-mode input subsystem, and a first keyboard and mouse sharing application, etc. The mobile phone includes a second camera, a second facial landmark detection module, a second integrated sensor information platform, a second multi-mode input subsystem, and a second keyboard and mouse sharing application, etc.

[0159] It is understandable that with keyboard and mouse sharing and hand-eye coordination enabled on both the PC and mobile phone, the PC and mobile phone can determine the user's facial orientation in real time or periodically. This allows for timely detection of whether the user's attention shifts between electronic devices, thereby better implementing the method described in this application's embodiments.

[0160] As shown in Figure 11, the method of this application embodiment mainly includes the following parts: (1) facial key point detection, (2) head pose estimation and calculation of the angle between the user's face and the electronic device, (3) face orientation judgment, and (4) keyboard and mouse crossing management. These parts will be described below.

[0161] (1) Facial landmark detection.

[0162] Taking the scenario where a user's face is captured on a PC instead of a mobile phone as an example, on one hand, the PC's first camera can capture the user's facial image and transmit it to the first facial landmark detection module. This module can then detect facial landmarks based on the captured image. On the other hand, the mobile phone's second camera can also capture the user's facial image and transmit it to the second facial landmark detection module. This module can also detect facial landmarks based on the captured image. Facial landmarks, also known as facial feature points, are a set of specific points identified in a facial image. These points correspond to facial features such as eyes, nose, mouth, and facial contours.

[0163] Taking the first facial landmark detection module as an example, in a possible implementation, the first facial landmark detection module can acquire images captured by a first camera. Optionally, this first camera can be a low-power camera (also known as a low-power sensor). The images captured by the low-power camera can be single-channel images, such as grayscale images, with relatively low resolution, but this does not affect the detection of facial landmarks. In this way, using a low-power camera can both reduce the power consumption of the electronic device and achieve the detection of facial landmarks.

[0164] After acquiring the image, the first facial landmark detection module can calculate facial landmarks based on the facial landmark detection model. These facial landmarks can be sparse features. Sparse features refer to a finite set of representative facial landmarks, or a relatively small number of facial landmarks. These landmarks are sufficient to describe the basic structure and features of the face. Thus, by calculating sparse features, the computational complexity of electronic devices can be reduced while maintaining an effective description of facial features. In other words, while maintaining recognition accuracy, the processing speed and recognition efficiency of electronic devices can also be improved.

[0165] In this embodiment, the sparse features may include six facial key points, such as the inner and outer corners of the eyes and the two corners of the mouth. Of course, the sparse features may include other facial key points, such as one or more of the following: the midpoint of the forehead, the inner corner of the eyebrow, the arch of the eyebrow, the tail of the eyebrow, the center of the pupil of the eye, the tip of the nose, the two sides of the nostrils, the midpoint of the upper lip, the midpoint of the lower lip, the chin, and the cheekbone. This embodiment does not limit the features.

[0166] In this embodiment, the facial landmark detection model can output a rotation matrix representing sparse features. This rotation matrix can be used to identify the changes in face coordinates relative to the coordinates of the first camera. For example, the rotation matrix can be a 3x4 matrix. The rotation matrix includes the three-dimensional (x-axis, y-axis, and z-axis) coordinates of the face with the first camera as the origin. The specific size and representation of the rotation matrix are not limited in this embodiment. After the facial landmark detection model outputs the rotation matrix, the first facial landmark detection module can pass the rotation matrix to the first integrated sensing information platform for head pose estimation.

[0167] Understandably, the implementation of the second facial landmark detection module is similar to that of the first facial landmark detection module, and will not be described in detail here. After obtaining the rotation matrix corresponding to the user's sparse features, the second facial landmark detection module can pass the rotation matrix to the second integrated sensor information platform for head pose estimation.

[0168] (2) Head pose estimation and calculation of the angle between the user's face and the electronic device.

[0169] For the first integrated sensing information platform, in a possible implementation, it can calculate the head pose in the world coordinate system based on the rotation matrix obtained from the first facial keypoint detection module. The head pose in the world coordinate system can be represented using Euler angles. That is, the first integrated sensing information platform can convert the rotation matrix into Euler angles. The specific conversion from rotation matrix to Euler angles can be found in relevant technologies and will not be elaborated further. Euler angles can include yaw angle, pitch angle, and roll angle. These three angles describe the rotation of the face around its own coordinate axes and can effectively describe the head pose in three-dimensional space.

[0170] The Yaw angle describes the rotation of a face around its vertical axis (z-axis), representing left or right rotation. A negative Yaw angle indicates leftward rotation, while a positive Yaw angle indicates rightward rotation. For example, a Yaw angle of 0° indicates the face is facing forward; a positive Yaw angle indicates rightward rotation; and a negative Yaw angle indicates leftward rotation.

[0171] The pitch angle describes the rotation of a face around its horizontal axis (y-axis), indicating whether the face is tilted forward or backward. A positive pitch angle occurs when the face is tilted upward, and a negative pitch angle occurs when the face is tilted downward.

[0172] The Roll angle describes the rotation of a face around its vertical axis (x-axis), representing the face rolling along a horizontal plane. The Roll angle is positive when the face rolls to the right and negative when it rolls to the left.

[0173] It is understandable that when a user shifts their attention between multiple electronic devices, it typically corresponds to a scenario where the user's face rotates to the left or right. Therefore, this embodiment of the application can calculate the user's facial orientation based on the yaw angle. This is more in line with the usage scenario of this embodiment of the application, making the calculation more accurate.

[0174] The first integrated sensor information platform can determine the user's face position and orientation based on head posture estimation, and then calculate the angle between the user's face orientation and the PC. The specific relationship between face position, face orientation, and angle can be found in the relevant information of the first angle in Table 1 below.

[0175] For example, as shown in Figure 12a, this is a schematic diagram illustrating the relationship between the angle of the user's face and the electronic device. In one possible scenario, the user's face is located to the right of the first camera and to the left of the second camera, and the user's face is turning from the phone towards the PC (i.e., the user is looking to the left). In this scenario, assuming the user's yaw angle is Yaw, the PC detects a distance of x1 between the first camera and the user's face along the x-axis, a distance of z1 between the first camera and the user's face along the z-axis, and the angle between the line connecting the first camera and the user's face and the user's face orientation is the first angle A1.

[0176] It is understood that in this embodiment, when the user's head is on the right side of the camera, as shown in Figure 12a relative to the first camera, the distance between the camera and the user's head in the x-axis direction can be set to a negative value. When the user's head is on the left side of the camera, as shown in Figure 12a relative to the second camera, the distance between the camera and the user's head in the x-axis direction can be set to a positive value. Of course, values ​​with opposite signs can also be used, and this embodiment does not limit this.

[0177] From the positional and triangular relationships in Figure 12a, it can be seen that the first included angle A1 satisfies the following formula: A1=arctan(|x1| / z1)-|Yaw|.

[0178] Since the user's face is located to the right of the first camera, the distance x1 between the first camera and the user's face along the x-axis is negative, meaning x1 is less than 0. However, the distance z1 between the first camera and the user's face along the z-axis is greater than 0, therefore arctan(x1 / z1) is less than 0. Because the user's face is looking to the left (facing PC), the Yaw angle is negative, meaning Yaw is less than 0. Removing the absolute value from the formula for the first included angle A1, the first included angle A1 satisfies the following formula: A1 = -arctan(x1 / z1) + Yaw.

[0179] Similar to the calculations performed on the first integrated sensor information platform, the second integrated sensor information platform can also calculate the head pose in the world coordinate system based on sparse features, and calculate the angle between the user's head and the mobile phone. The specific relationship between the face position, face orientation, and angle can be found in the relevant information for the second angle in Table 1 below.

[0180] As shown in Figure 12a, when the user's face is located to the left of the second camera and the user's face is turning from the phone to the PC, assuming the user's yaw angle is Yaw, the phone detects a distance of x2 between the second camera and the user's face along the x-axis, and a distance of z2 between the second camera and the user's face along the z-axis. The angle between the line connecting the second camera and the user's face and the direction the user's face is facing is the second angle B1. From the trigonometric relationships in Figure 12a, it can be seen that the second angle B1 satisfies the following formula: B1=arctan(|x2| / z2)+|Yaw|.

[0181] Since the user's face is located to the left of the second camera, the distance x2 between the second camera and the user's face along the x-axis is positive, meaning x2 is greater than 0. The distance z2 between the second camera and the user's face along the z-axis is also greater than 0, therefore arctan(x2 / z2) is greater than 0. Because the user's face is looking to the left (facing PC), the Yaw angle is negative, meaning Yaw is less than 0. Removing the absolute value from the formula for the second included angle B1, the second included angle B1 satisfies the following formula: B1 = arctan(x2 / z2) - Yaw.

[0182] In another possible scenario, the user's face can be turned from the PC to the phone, with the user's face positioned to the right of the first camera and to the left of the second camera. Then the first included angle A2 satisfies the following formula: A2 = arctan(|x1| / z1) + |Yaw|.

[0183] Since the user's face is located to the right of the first camera, the distance x1 between the first camera and the user's face along the x-axis is negative, meaning x1 is less than 0. However, the distance z1 between the first camera and the user's face along the z-axis is greater than 0, therefore arctan(x1 / z1) is less than 0. Because the user's face is facing right, i.e., towards the phone, the Yaw angle is positive, meaning Yaw is greater than 0. Therefore, removing the absolute value from the formula for the first included angle A2, the first included angle A2 satisfies the following formula: A2 = -arctan(x1 / z1) + Yaw.

[0184] The second included angle B2 satisfies the following formula: B2=arctan(|x2| / z2)-|Yaw|.

[0185] Since the user's face is located to the left of the second camera, the distance x2 between the second camera and the user's face along the x-axis is positive, meaning x2 > 0. The distance z2 between the second camera and the user's face along the z-axis is also greater than 0, therefore arctan(x2 / z2) is greater than 0. Because the user's face is facing right (towards the phone), the Yaw angle is positive, meaning Yaw > 0. Removing the absolute value from the formula for the second included angle B2, the second included angle B2 satisfies the following formula: B2 = arctan(x2 / z2) - Yaw.

[0186] Table 1

[0187] It's understandable that the PC and phone might not be on the same horizontal line, as shown in Figure 12b, but the calculation method is similar to that in Figure 12a, so it won't be repeated. It's important to note that because the PC and phone are not on the same horizontal line, the Yaw angle calculated by the PC and the phone will be different. For example, in Figure 12b, the PC calculates Yaw angle 1, while the phone calculates Yaw angle 2. Therefore, when calculating the first included angle, the PC needs to use Yaw 1. When calculating the second included angle, the phone needs to use Yaw 2.

[0188] (3) Judging by the direction of the face.

[0189] Since the PC is the master device and the mobile phone is the slave device, the second included angle calculated by the mobile phone needs to be sent to the PC, which then judges the first and second included angles. Therefore, after the second integrated sensor information platform determines the second included angle between the mobile phone and the user's face, it can instruct the mobile phone to send the second included angle to the PC. In this embodiment, the PC and the mobile phone can communicate and transmit data based on a distributed soft bus. Optionally, when the mobile phone sends the second included angle to the PC, it can also send information such as the mobile phone's device identifier. That is, when the slave device sends the included angle information to the master device, it can also send the slave device's device identifier, such as device ID. This makes it easier for the master device to distinguish between the multiple included angles reported and the correspondence between different slave devices, facilitating subsequent focus switching control.

[0190] After the PC obtains the second angle from the mobile phone, the first integrated sensor information platform can determine the user's face orientation based on the first and second angles.

[0191] If the first included angle is smaller than the second included angle, the first integrated sensor information platform can determine that the user's face is facing the PC.

[0192] If the first included angle is greater than or equal to the second included angle, the first integrated sensor information platform can determine that the user's face is facing the mobile phone, which can also be understood as the focus of the input device still being on the mobile phone screen, and no focus switching is required.

[0193] (4) Keyboard and mouse crossover management.

[0194] When the first included angle is smaller than the second included angle, and the user's face is towards the PC, the first integrated sensing information platform can transmit instruction information to the first keyboard and mouse sharing application. This instruction information can be used to instruct the first keyboard and mouse sharing application to display focus switching information. After receiving the instruction information, the first keyboard and mouse sharing application can display focus switching prompt information on the PC interface. The specific method of displaying focus switching prompt information on the PC interface can be found in the description of prompt information 401 in the corresponding embodiment of Figure 4, and will not be repeated here.

[0195] In response to a user pressing the Ctrl key, the first multi-modal input subsystem can send a keyboard and mouse traversal request to the first keyboard and mouse sharing application. The first keyboard and mouse sharing application can then send a keyboard and mouse traversal request to the second keyboard and mouse sharing application based on information such as the phone's device identifier. Upon receiving the keyboard and mouse traversal request, the second keyboard and mouse sharing application can then forward the request to the second multi-modal input subsystem.

[0196] After receiving a keyboard and mouse traversal request, the second multi-modal input subsystem can transmit an instruction to the second keyboard and mouse sharing application. This instruction instructs the second keyboard and mouse sharing application to hide the keyboard and mouse focus. Upon receiving this instruction, the second keyboard and mouse sharing application can hide the keyboard and mouse focus on the phone screen. After the focus is hidden, the second multi-modal input subsystem can return a response to the keyboard and mouse traversal request (referred to as a request response) to the second keyboard and mouse sharing application. The second keyboard and mouse sharing application can then further return a request response to the first keyboard and mouse sharing application. Upon receiving this request response, the first keyboard and mouse sharing application can transmit the request response back to the first multi-modal input subsystem.

[0197] After receiving a request response, the first multi-modal input subsystem can transmit instruction information to the first keyboard and mouse sharing application. This instruction information instructs the first keyboard and mouse sharing application to display the keyboard and mouse focus. Upon receiving this instruction information, the first keyboard and mouse sharing application can display the keyboard and mouse focus on the PC interface. The specific method of displaying the keyboard and mouse focus on the PC can be described in the relevant embodiments of Figures 5 and 6, and will not be repeated here.

[0198] When the first included angle is greater than or equal to the second included angle, the user's face is facing the phone, and no focus switching is required. In a possible scenario, if the user's face turns from the PC to the phone, the first integrated sensor information platform can send an instruction to the second integrated sensor information platform to instruct the second keyboard and mouse sharing application to display focus switching information. The specific method of displaying the focus switching prompt on the phone interface can be found in the description of prompt information 401 in the corresponding embodiment of Figure 4, and will not be repeated here.

[0199] As described in the embodiments corresponding to Figures 11 and 12, implementing the method of this application requires determining the user's face orientation based on multiple electronic devices. As shown in Figure 13, taking a first electronic device (e.g., a PC) and a second electronic device (e.g., a mobile phone) as examples, the first and second electronic devices need to acquire images containing the user's face, calculate facial key points, and perform head pose estimation, respectively. Thus, the first electronic device can obtain a first angle with the user's face, and the second electronic device can obtain a second angle with the user's face. Taking the first electronic device as the master device and the second electronic device as the slave device, the second electronic device can send the second angle to the first electronic device. The second electronic device can determine the user's face orientation based on the relationship between the first and second angles. Optionally, the determination of the first and second angles can be performed by the slave device (i.e., the second electronic device). In other words, the process of determining the user's face orientation based on multiple angles can be performed by any one of multiple keyboard and mouse sharing devices; this application embodiment does not limit this.

[0200] In possible implementations, the attention detection method based on face orientation of this application embodiment can also be applied to other use cases. Some possible extended scenarios are exemplified below. In practical applications, the method of this application embodiment can be applied to even more scenarios, without limitation.

[0201] Extended Scenario 1: Application Continuation Scenario.

[0202] In possible scenarios, application continuity functionality can be implemented between multiple electronic devices. Application continuity functionality can be understood as the ability to seamlessly switch between and continue using a particular application across multiple electronic devices. For example, taking a PC and a mobile phone as examples, as shown in Figure 14a, the mobile phone can display a document application editing interface 1400, where the user can edit document content. This application embodiment does not limit the specific document content.

[0203] When the mobile phone and PC are close together, as shown in Figure 14b, a connection icon 1402 can be displayed on the PC's interface 1401. For example, the PC can display the connection icon 1402 in the taskbar, shortcut window, or other interface entry point of interface 1401; this embodiment does not limit this. The connection icon 1402 is used to prompt the user to continue using the document application via the PC. The connection icon 1402 may include, but is not limited to, one or more of the following: the document application's application icon, the mobile phone's device type icon, or the mobile phone's device type name. For example, the connection icon 1402 may display the mobile phone's device type icon 1403 in the upper right corner of the document application icon. The specific display style of the connection icon 1402 is not limited in this embodiment.

[0204] In response to the user triggering the continuation icon 1402, as shown in Figure 14c, the document application's editing interface 1404 can be displayed on the PC's interface 1401. It is understood that the content and editing progress in the PC's editing interface 1404 are the same as those in the mobile phone's editing interface 1400. This allows users to conveniently continue editing document content on the PC.

[0205] To better prompt the user, the PC can play an animation effect of the continuation icon 1402, such as the PC displaying the continuation icon 1402 in a flashing manner for a period of time. The specific display style of the animation effect is not limited in this embodiment.

[0206] However, in some scenarios, when the animation effect of the application continuation icon 1402 is playing on the PC, the user's attention may still be on the mobile phone. By the time the user shifts their attention from the mobile phone to the PC, the animation effect of the continuation icon 1402 on the PC has already finished playing. This may cause the user to miss the application continuation prompt, making it difficult for the user to use the application continuation function effectively on the PC and reducing the user experience.

[0207] In this embodiment, by combining a face orientation attention detection method, the PC can first determine the user's face orientation when displaying the application continuation icon 1402. When the PC determines that the user's face is facing the PC, i.e., when the user's attention shifts from the phone to the PC, the PC then begins displaying the animation effect of the application continuation icon 1402. This allows the user to notice the application continuation prompt in a timely manner, enabling the user to better utilize the application continuation function on the PC, thereby improving the user experience.

[0208] Extended Scenario 2: Scenarios involving protecting screen content.

[0209] In possible scenarios, users can use electronic devices to edit documents, watch videos, and chat on social media. Taking a PC as an example, as shown in Figure 15a, the PC's interface 1500 can display a document interface 1501, where users can edit or view documents. When someone else approaches the user, for example, to communicate with them, the PC can detect the user's head orientation to determine whether the content displayed on interface 1500 needs to be adjusted, or whether the content on interface 1500 should not be displayed.

[0210] In a possible implementation, as shown in Figure 15b, when the user is not looking at the PC, the PC can detect that the angle formed by the camera and the user's head orientation is greater than or equal to a preset angle threshold. The PC can then adjust the content displayed on interface 1500. Adjusting the content displayed on interface 1500 can include, but is not limited to, the following display methods: adding a mask layer on interface 1500 (as shown in Figure 15c), blurring interface 1500, or displaying preset content on interface 1500. The preset content can include content pre-set by the electronic device or the user (such as preset images).

[0211] Alternatively, when the angle formed by the PC's detection camera and the user's head orientation is greater than or equal to a preset angle threshold, the PC's interface 1500 may not display the document interface 1501. Not displaying the document interface 1501 may include, but is not limited to, the following display methods: minimizing the document interface 1501, closing the document application, displaying the PC's desktop, displaying the PC's lock screen (as shown in Figure 15d), or turning off the PC screen.

[0212] In this way, adjusting the displayed content of the interface or not displaying the interface can make the displayed content blurry or obscured, making it impossible for others to clearly identify the content displayed on the interface, thus better protecting the user's information.

[0213] Optionally, in the PC's screen-off scenario, the PC can turn off the screen if it detects that no user-triggered operation has been received within a preset time period. Screen off, also known as screen dimming, can be understood as the PC screen being turned off or dimmed to save battery power or prevent accidental touches. After the screen is turned off, the PC screen no longer displays content, but the PC may still be running.

[0214] However, in some scenarios, even though the user hasn't triggered any action, they may still be viewing the content displayed on interface 1500, such as when viewing content on document interface 1501. In this scenario, if the PC screen turns off, it will affect the user's document viewing experience, thus degrading the user experience.

[0215] Therefore, in this embodiment, if the electronic device does not receive any user-triggered operation within a preset time period, the electronic device can also determine the user's face orientation. If the user's face is not facing the electronic device within the preset time period, it indicates that the user is not viewing the content displayed on the screen, and the electronic device can turn off the screen. If the user's face is facing the electronic device within the preset time period, it indicates that the user may be viewing the content displayed on the screen, and the electronic device will not turn off the screen. In this way, the electronic device can more accurately determine the timing of screen turning off, thereby improving the user experience. It is understood that the above optimization for the screen-off scenario also applies to the lock screen scenario, and will not be elaborated further.

[0216] Extended Scenario 3: Call Scenario.

[0217] In possible scenarios, users can use electronic devices to make voice or video calls. Taking a PC as an example, as shown in Figure 16a, the PC's interface 1600 can display a call interface 1601. The call interface 1601 can include the call duration, the other party's avatar, the other party's name, a microphone control 1602, a hang-up control, and a speaker control. During the call, the microphone control 1602 is in the on state (as shown in Figure 16a), and the electronic device can send the user's audio data to the other party's electronic device. When someone else is near the user, for example, when communicating with the user, in order to prevent the other party from hearing the user's conversation with others, the electronic device can detect the user's head orientation to determine whether to mute the call.

[0218] In a possible implementation, as shown in Figure 16b, when the user is not looking at the PC, the PC can detect that the angle formed by the camera 1603 and the user's head orientation is greater than or equal to a preset angle threshold. The PC can then set the microphone control 1602 to a call mute state (as shown in Figure 16c). This call mute state can include the electronic device not using audio data, and also the electronic device not sending the collected audio data to the other party's electronic device. This allows for reasonable protection of the user's information based on their head orientation, thereby improving the user experience.

[0219] Figure 17 illustrates a schematic diagram of an attention detection method based on face orientation according to an embodiment of this application. The method is applied to a first electronic device and includes:

[0220] S1701. Determine the first angle between the first electronic device and the user's face, and obtain the second angle from the second electronic device; wherein the second angle is the angle between the second electronic device and the user's face, and the second electronic device responds to the input event of the input device.

[0221] In this embodiment, the first electronic device can be any electronic device with keyboard and mouse sharing functionality and hand-eye coordination functionality. Furthermore, when the first electronic device and other devices implement the method of this embodiment, the first electronic device can act as the master device. The master device can be understood as the initiator of the keyboard and mouse sharing functionality, and one or more devices capable of keyboard and mouse sharing can be selected within the master device. The master device can also receive and process data from other keyboard and mouse sharing devices. For example, the first electronic device can include the PC in the above embodiments.

[0222] The second electronic device can be any electronic device with keyboard and mouse sharing functionality and hand-eye coordination functionality. Furthermore, when the second electronic device and other devices implement the methods of the embodiments of this application, the second electronic device can act as a slave device (or secondary device). The slave device can send relevant information such as the second included angle and device identifier to the master device. For example, the second electronic device can include the mobile phone in the above embodiments.

[0223] Input devices may include devices such as keyboards, mice, or touchscreens that can be shared among multiple electronic devices. It is understood that when a second electronic device responds to an input event from an input device, the interface of the second electronic device may also display the focus of the input device. This focus may include a cursor or mouse pointer.

[0224] The specific method for determining the first angle between the first electronic device and the user's face can be found in the head pose estimation of the corresponding embodiment in Figure 11 and the relevant description in the calculation of the angle between the face and the device, which will not be repeated here.

[0225] S1702, when the first included angle is less than the second included angle, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device.

[0226] In this embodiment, the first included angle is smaller than the second included angle, indicating that the angle between the user's face and the first electronic device is smaller, while the angle between the user's face and the second electronic device is larger, meaning the user's face is facing the first electronic device. In this case, the electronic device responding to the input event of the input device switches from the second electronic device to the first electronic device, which can also be understood as the focus of the input device switching from the second electronic device to the first electronic device. For the specific focus switching method, please refer to the relevant description in (4) of the corresponding embodiment of Figure 11 for keyboard and mouse crossing management, which will not be repeated here.

[0227] Understandably, if the first included angle is greater than or equal to the second included angle, it means the user's face is facing the second electronic device. In other words, the user's attention is still on the second electronic device, and the user's face has not turned towards the first electronic device. Therefore, in this case, the input device's focus remains on the second electronic device; that is, the electronic device responding to the input event is still the second electronic device, and there is no need to switch to the first electronic device.

[0228] S1703. In response to the user triggering a first operation on the input device, execute the input event corresponding to the first operation.

[0229] In a possible scenario, after step S1703, if the user's face is turned towards the second electronic device again, the electronic device responding to the input event of the input device can be switched from the first electronic device to the second electronic device. In response to the user triggering an operation on the input device, the second electronic device executes the input event corresponding to that operation.

[0230] In this embodiment, multiple electronic devices sharing an input device (including a first electronic device and a second electronic device) can calculate the angles (including a first angle and a second angle) with the user's face, and comprehensively determine the user's face orientation, thereby controlling the switching of the input device's focus on different electronic devices. In this way, by determining the face orientation, the focus of the input device can be quickly switched on different electronic devices, eliminating the need for the user to cumbersomely switch the focus by clicking a mouse or screen, thus improving the user experience.

[0231] Optionally, based on the embodiment corresponding to FIG17, after the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, it may further include: the interface of the first electronic device displays a first prompt message, the first prompt message being used to instruct the first electronic device to respond to the input event of the input device.

[0232] In this embodiment, the first prompt information can also be understood as a prompt information indicating a focus switch (from the second electronic device to the first electronic device). The first prompt information may include, but is not limited to, one or more of the following: icon-type prompt information, text-type prompt information, and mouse tooltips. Mouse tooltips may include mouse cursor tooltips and mouse pointer tooltips. Mouse pointer tooltips may include animation effects of the mouse pointer and / or animation effects of water ripples around the mouse pointer. Specific methods of mouse pointer tooltips can be referred to the relevant descriptions in the corresponding embodiments of Figure 5 and Figure 6, and will not be repeated here.

[0233] Understandably, primary information such as mouse cursor tooltips allows users to quickly understand that they have switched the keyboard and mouse focus from the secondary electronic device to the primary one. Mouse cursor tooltips also help users quickly locate the mouse pointer, thus improving the user experience.

[0234] Optionally, based on the embodiment corresponding to Figure 17, the first prompt information includes one or more of the following types of prompt information: icon-type prompt information, text-type prompt information, and mouse tooltip.

[0235] It is understood that the explanation of the first prompt message can refer to the relevant description of the first prompt message in the above optional embodiments, and will not be repeated here.

[0236] In this embodiment, icon-based prompts convey information through visual symbols, making it easier for users to identify and understand the meaning of the first prompt. Text-based prompts help users better understand which electronic device is currently the focus of the input device, making the interface more interactive and user-friendly. Mouse prompts allow users to quickly locate the mouse pointer or cursor, facilitating further editing or other operations. In other words, through the first prompt, users can promptly understand that the focus of the keyboard and mouse has been switched from the second electronic device to the first electronic device, thereby improving the user experience of using the hand-eye coordination function.

[0237] Optionally, based on the embodiment corresponding to Figure 17, the mouse tooltip includes an animation effect of the mouse pointer displayed on the interface of the first electronic device. The animation effect of the mouse pointer includes one or more of the following animation effects: changes in the size of the mouse pointer, changes in the color of the mouse pointer, changes in the glowing effect of the edge of the mouse pointer, and changes around the mouse pointer.

[0238] In this embodiment, the animation effect of the mouse pointer can be referred to the relevant description in the corresponding embodiment of Figure 5, and will not be repeated here. The color change of the mouse pointer may include changes in color hue, color brightness, and / or color transparency, etc., which are not limited in this embodiment.

[0239] Changes around the mouse pointer can include a water ripple animation effect around the mouse pointer. For specific water ripple animation effects around the mouse pointer, please refer to the relevant description in the corresponding embodiment of Figure 6, which will not be repeated here.

[0240] Understandably, by using mouse pointer animations and water ripple animations as prompts for focus switching, users can promptly understand that the focus of the keyboard and mouse has been switched to the first electronic device, thus making it easier for users to quickly locate the mouse pointer and improving the user experience.

[0241] Optionally, based on the embodiment corresponding to Figure 17, before the electronic device responding to the input event of the input device switches from the second electronic device to the first electronic device, it may further include: the interface of the first electronic device displays a second prompt message, the second prompt message being used to indicate that by triggering a second operation, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device; the electronic device responding to the input event of the input device switching from the second electronic device to the first electronic device includes: in response to the second operation, the electronic device responding to the input event of the input device switches from the second electronic device to the first electronic device.

[0242] In this embodiment, the second prompt information can also be understood as indicating to the user that the user is currently looking at the first electronic device, and can also be used to instruct the user to switch focus. The second prompt information may include icon prompts and / or text prompts. For example, the second prompt information may be prompt information 401 in the embodiment corresponding to Figure 4.

[0243] The second operation may include key operations on the input device, wherein the key may include the Ctrl key, Enter key, Shift key, space key, or clicking the mouse, etc., and the embodiments of this application are not limited thereto.

[0244] Understandably, the second prompt helps users quickly understand how to switch input device focus between different electronic devices, improving operational accuracy and enabling users to become more proficient in using keyboard and mouse sharing and hand-eye coordination functions.

[0245] Optionally, based on the embodiment corresponding to Figure 17, the second operation includes key operations on the input device.

[0246] It is understood that the explanation of the second operation can refer to the relevant description of the second operation in the above optional embodiments, and will not be repeated here.

[0247] In this embodiment, the first electronic device responds to input events from the input device by having the user trigger a second operation. This allows the user to choose whether to switch the focus of the input device according to their needs, increasing the user's selection and settings of certain functions within the electronic device, enhancing interactivity, and thus improving the user experience.

[0248] Optionally, based on the embodiment corresponding to Figure 17, before determining the first angle between the first electronic device and the user's face, and before obtaining the second angle from the second electronic device, the method may further include: receiving an operation to set the second electronic device as an electronic device that shares an input device with the first electronic device.

[0249] In this embodiment, receiving the operation of setting the second electronic device as an electronic device that shares the input device with the first electronic device can also be understood as setting the second electronic device as a slave device, or as adding a second electronic device for keyboard and mouse sharing to the first electronic device. For example, taking a PC as the first electronic device and a tablet or mobile phone as the second electronic device, this operation can include the user triggering the tablet option 8031 ​​or the user triggering the mobile phone option 8032 in the embodiment corresponding to FIG8.

[0250] By configuring the second electronic device to share an input device with the first electronic device, keyboard and mouse sharing and hand-eye coordination can be achieved between the two devices. This allows users to move the mouse or share input devices between multiple electronic devices, resulting in smoother cross-screen operations.

[0251] Optionally, based on the embodiment corresponding to FIG17, the input device is a device included in the first electronic device, or the input device is a device independent of the first electronic device.

[0252] In this embodiment, the input device is a device included in the first electronic device, or it can be understood as a built-in device within the first electronic device. The built-in device can be tightly integrated with other parts or modules of the first electronic device, eliminating the need for additional input devices or additional settings or connections, thus offering convenience and speed. For example, if the first electronic device is a laptop computer, the input device could be the laptop's keyboard, touchpad, etc.

[0253] An input device is a device independent of the first electronic device; it can also be understood as an external device of the first electronic device. Users can choose the type of input device that suits their personal preferences, and they can also adjust the position and angle of the external device as needed, providing a more flexible input environment.

[0254] Optionally, based on the embodiment corresponding to Figure 17, the input device includes a keyboard or a mouse.

[0255] Optionally, based on the embodiment corresponding to Figure 17, the first electronic device includes a first camera, the second electronic device includes a second camera, the first included angle is the angle between the first camera and the direction the face is facing, and the second included angle is the angle between the second camera and the direction the face is facing.

[0256] In this embodiment, the first included angle may include the first included angle A1 or the first included angle A2 in the embodiment corresponding to FIG11, and the second included angle may include the second included angle B1 or the second included angle B2 in the embodiment corresponding to FIG11. The specific details of the first and second included angles can be found in the relevant descriptions of the embodiments corresponding to FIG11 and FIG12, and will not be repeated here.

[0257] Understandably, by considering the angles (including the first and second angles) associated with multiple electronic devices, a user's face orientation can be comprehensively determined. Compared to determining the user's face orientation based on a single electronic device, determining the orientation based on the angles of multiple electronic devices can reduce data fluctuations and inaccurate calculation results, thereby reducing judgment errors and more accurately controlling the switching of the input device's focus on different electronic devices.

[0258] Optionally, based on the embodiment corresponding to Figure 17, the method may further include: the first electronic device displaying a first interface; determining a third angle between the first electronic device and the user's face; if the third angle is greater than or equal to a preset angle threshold, adjusting the display content of the first interface, or not displaying the first interface.

[0259] In this embodiment of the application, the first interface may include interface 1500 in the embodiment corresponding to FIG15.

[0260] The third included angle being greater than or equal to a preset angle threshold can be understood as a scenario where the user is not looking at the first electronic device. The preset angle threshold can be a preset angle in the first electronic device, for example, the preset angle threshold can be greater than or equal to 45°, etc. The specific value of the preset angle threshold is not limited in the embodiments of this application.

[0261] The adjustment of the displayed content of the first interface can be referred to the relevant description of adjusting the content displayed by interface 1500 in the corresponding embodiment of Figure 15; the non-display of the first interface can be referred to the relevant description of non-displaying document interface 1501 in the corresponding embodiment of Figure 15, and will not be repeated here. It is understood that adjusting the displayed content of the first interface or not displaying the first interface can make the content displayed on the first interface blurry or obscured, making it impossible for others to clearly identify the content displayed on the first interface, thereby better protecting the user's information.

[0262] Optionally, based on the embodiment corresponding to Figure 17, the method may further include: the first electronic device displaying the second interface; determining the fourth angle between the first electronic device and the user's face; if the duration for which the fourth angle is greater than or equal to a preset angle threshold continues to exceed a preset duration, entering a screen-off state or a screen-locked state.

[0263] In this embodiment, the preset duration can be understood as the duration during which the first electronic device determines that it has not received any user-triggered operation before entering a screen-off or screen-locked state. This preset duration can be pre-set by the electronic device or by the user; this embodiment does not impose any limitations on this.

[0264] If the electronic device does not receive any user-triggered actions within a preset time period, it can also determine the user's facial orientation. If the user's face is not facing the electronic device within this preset time period, it indicates that the user is not viewing the content displayed on the screen, and the electronic device can turn off or lock the screen. If the user's face is facing the electronic device within this preset time period, it indicates that the user may be viewing the content displayed on the screen, and the electronic device will not turn off or lock the screen. In this way, the electronic device can more accurately determine when to turn off or lock the screen, thereby improving the user experience.

[0265] Optionally, based on the embodiment corresponding to Figure 17, the method may further include: the first electronic device displaying a call interface; determining a fifth angle between the first electronic device and the user's face; and, if the fifth angle is greater than or equal to a preset angle threshold, the first electronic device activating a call mute function.

[0266] In this embodiment of the application, the call interface may include the call interface 1601 in the embodiment corresponding to FIG16.

[0267] Taking the embodiment corresponding to Figure 16 as an example, when the first electronic device activates the call mute function, it can be understood that the first electronic device can set the microphone control 1602 to a call mute state (as shown in c of Figure 16). At this time, the microphone control 1602 is in a closed state. The call mute state can include the electronic device not using audio data, and also includes the electronic device not sending the collected audio data to the other party's electronic device. This allows for reasonable protection of user information based on the user's head orientation, thereby improving the user experience.

[0268] Understandably, when the fifth included angle is less than a preset angle threshold, the first electronic device can disable the call mute function. At this time, the microphone control 1602 is in the open state (as shown in Figure 16a).

[0269] Optionally, based on the embodiment corresponding to Figure 17, the method may further include: establishing a communication connection between the first electronic device and the third electronic device, wherein the third electronic device displays an interface of the first application; if the distance between the first electronic device and the third electronic device is less than a preset distance and the sixth angle is less than a preset angle threshold, the first electronic device displays a third prompt message, wherein the third prompt message includes the icon of the first application; wherein the sixth angle is the angle between the first electronic device and the user's face.

[0270] In this embodiment, the third electronic device can be understood as any electronic device capable of connecting with the first electronic device. After the first electronic device and the third electronic device establish a communication connection, application connection can be achieved. The interface of the first application may include the document application editing interface 1400 in the embodiment corresponding to Figure 14. Taking the embodiment corresponding to Figure 14 as an example, the first electronic device can be a PC, and the third electronic device can be a mobile phone.

[0271] The preset distance can be understood as the distance that enables the first and third electronic devices to achieve application continuity. The third prompt message can be understood as the continuity icon 1402 or the animation effect of the continuity icon 1402.

[0272] Understandably, when displaying the third prompt, the first electronic device can first determine the user's face orientation. Once the first electronic device confirms that the user's face is facing it, meaning the user's attention has shifted from the third electronic device to the first, it then begins displaying the third prompt. This allows the user to promptly notice the application continuation prompt, enabling them to better utilize the application continuation function on the first electronic device, thereby improving the user experience.

[0273] Optionally, based on the embodiment corresponding to Figure 17, before determining the first angle between the first electronic device and the user's face, the method may further include: acquiring an image containing the user's face; determining key point information of the user's face based on the image; determining the user's head posture information based on the key point information, the head posture information including the yaw angle of the user's head; and determining the first angle between the first electronic device and the user's face, including: determining the first angle based on the user's head posture information and the distance between the camera and the user's face in the horizontal and vertical directions.

[0274] In this embodiment, facial key points can be represented by sparse features of the user's face. The determination of key point information of the user's face based on the image can be referred to the relevant description in facial key point detection of embodiment (1) in Figure 11, which will not be repeated here. Facial key points may include the inner and outer corners of both eyes, and the two corners of the mouth. The specific facial feature information included in the facial key points is not limited in this embodiment.

[0275] The user's head pose information and the first included angle are determined based on key point information. Refer to the description in (2) of the corresponding embodiment in Figure 11 for head pose estimation and the calculation of the angle between the face and the device; these details will not be repeated here. The yaw angle of the user's head can be understood as the rotation angle of the user's head around its own coordinate axis (such as the z-axis), used to represent the left and right rotation of the user's head. Refer to the relevant description in the corresponding embodiment in Figure 11 for details. It is understood that sparse features are a relatively small number of facial key points, sufficient to describe the basic structure and features of the face. By calculating sparse features, the computational complexity of the electronic device can be reduced while maintaining an effective description of facial features, thus improving the processing speed and recognition efficiency of the electronic device while maintaining recognition accuracy.

[0276] The above detailed embodiments have provided a detailed explanation of the purpose, technical solutions, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific implementations of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the embodiments of this application should be included within the scope of protection of the embodiments of this application. The module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module; no specific restrictions are placed on the module names.

[0277] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0278] The foregoing mainly describes the technical solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that the method steps described in conjunction with the embodiments of this application can be implemented in hardware or a combination of hardware and software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0279] This application provides an electronic device that may include one or more processors and a memory. The memory is coupled to one or more processors. The memory may be used to store computer program code, which includes computer instructions. The one or more processors may invoke the computer instructions to cause the electronic device to execute the technical solutions described above.

[0280] This application provides a chip system. This chip system is applied to an electronic device and may include one or more processors. The one or more processors can be used to invoke instructions to cause the electronic device to execute the technical solutions described above.

[0281] Figure 18 is a schematic diagram of a chip system provided in an embodiment of this application. The chip system 1800 includes one or more processors 1801, communication lines 1802, communication interfaces 1803, and memory 1804.

[0282] In some implementations, memory 1804 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0283] The methods described in the embodiments of this application can be applied to, or implemented by, processor 1801. Processor 1801 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1801 or by instructions in software form. Processor 1801 may be a general-purpose processor (e.g., a microprocessor or conventional processor), DSP, application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (OPG) or other programmable logic device, discrete gate, transistor logic device, or discrete hardware component. Processor 1801 can implement or execute the processing-related methods, steps, and logic block diagrams in the embodiments of this application.

[0284] The method described in this application embodiment can be executed using a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules can be located in the memory 1804, and the processor 1801 can read information from the memory 1804 and, in conjunction with its hardware, complete the steps of the above method. The processor 1801, the memory 1804, and the communication interface 1803 can communicate via the communication line 1802.

[0285] This application also provides a readable storage medium (also referred to as a computer-readable storage medium). The readable storage medium includes computer instructions. When these computer instructions are executed on an electronic device, they cause the electronic device to perform the technical solutions described in the above embodiments.

[0286] The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored or transmitted on a readable medium as one or more instructions or programs. The readable medium can include storage media and communication media, and can also include any medium that can transfer a program from one place to another. The storage medium can be any accessible target medium.

[0287] In possible implementations, the readable medium may include random access memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, optical disc read-only memory, or other optical disc storage. The readable medium may also include disk storage or other disk storage devices, or be directed to any other medium or store the required program in the form of instructions or data structures, and be accessible.

[0288] This application also provides a program product (also referred to as a computer program product). The program product includes computer program code. When the computer program code runs on an electronic device, it causes the electronic device to execute the technical solutions described in the above embodiments.

[0289] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions between the various embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0290] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An attention detection method based on facial orientation, characterized in that, The method is applied to a first electronic device, and the method includes: Determine a first angle between the first electronic device and the user's face, and obtain a second angle from the second electronic device; wherein the second angle is the angle between the second electronic device and the user's face, and the second electronic device responds to an input event from an input device; When the first included angle is smaller than the second included angle, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device; In response to a user triggering a first operation on the input device, the input event corresponding to the first operation is executed.

2. The method as described in claim 1, characterized in that, After the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, it further includes: The interface of the first electronic device displays a first prompt message, which is used to instruct the first electronic device to respond to the input event of the input device.

3. The method as described in claim 2, characterized in that, The first prompt message includes one or more of the following types of prompt messages: icon-type prompt messages, text-type prompt messages, and mouse tooltips.

4. The method as described in claim 3, characterized in that, The mouse tooltip includes an animation effect of the mouse pointer displayed on the interface of the first electronic device. The animation effect of the mouse pointer includes one or more of the following: changes in the size of the mouse pointer, changes in the color of the mouse pointer, changes in the glowing effect of the edge of the mouse pointer, and changes around the mouse pointer.

5. The method according to any one of claims 1-4, characterized in that, Before the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, it further includes: The interface of the first electronic device displays a second prompt message, which indicates that by triggering a second operation, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device. The electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, including: In response to the second operation, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device.

6. The method as described in claim 5, characterized in that, The second operation includes key operations on the input device.

7. The method as described in claim 6, characterized in that, The key operations include any of the following: pressing the Ctrl key, pressing the Enter key, pressing the Shift key, pressing the space bar, or clicking the mouse.

8. The method according to any one of claims 1-7, characterized in that, Before determining the first angle between the first electronic device and the user's face, and before obtaining the second angle from the second electronic device, the method further includes: The system receives an operation to configure the second electronic device as an electronic device that shares the input device with the first electronic device.

9. The method according to any one of claims 1-8, characterized in that, The input device is either a device included in the first electronic device or a device independent of the first electronic device.

10. The method according to any one of claims 1-9, characterized in that, The input device includes a keyboard or a mouse.

11. The method according to any one of claims 1-10, characterized in that, The first electronic device includes a first camera, the second electronic device includes a second camera, the first included angle is the angle between the first camera and the direction the face is facing, and the second included angle is the angle between the second camera and the direction the face is facing.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: The first electronic device displays the first interface; Determine the third angle between the first electronic device and the user's face; If the third included angle is greater than or equal to a preset angle threshold, adjust the displayed content of the first interface, or do not display the first interface.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: The first electronic device displays the second interface; Determine the fourth angle between the first electronic device and the user's face; If the duration for which the fourth included angle is greater than or equal to a preset angle threshold continues to exceed a preset duration, the screen will enter an off state or a locked state.

14. The method according to any one of claims 1-13, characterized in that, The method further includes: The first electronic device displays the call interface; Determine the fifth angle between the first electronic device and the user's face; When the fifth included angle is greater than or equal to a preset angle threshold, the first electronic device activates the call mute function.

15. The method according to any one of claims 1-14, characterized in that, The method further includes: The first electronic device establishes a communication connection with the third electronic device, and the third electronic device displays the interface of the first application; If the distance between the first electronic device and the third electronic device is less than a preset distance, and the sixth included angle is less than a preset angle threshold, the first electronic device displays a third prompt message, which includes the icon of the first application; wherein, the sixth included angle is the angle between the first electronic device and the user's face.

16. The method according to any one of claims 1-15, characterized in that, Before determining the first angle between the first electronic device and the user's face, the method further includes: Acquire an image containing the user's face, and determine key point information of the user's face based on the image; The user's head posture information is determined based on the key point information, and the head posture information includes the yaw angle of the user's head; Determining the first angle between the first electronic device and the user's face includes: The first included angle is determined based on the user's head posture information and the distance between the camera and the user's face in the horizontal and vertical directions.

17. An attention detection method based on facial orientation, characterized in that, The method is applied to a second electronic device, and the method includes: In response to a third operation triggered by the user on the input device, the input event corresponding to the third operation is executed; Determine a second angle between the second electronic device and the user's face, and send the second angle to the first electronic device; If the first included angle is less than the second included angle, the fourth operation triggered by the user on the input device will not be responded to, wherein the first included angle is the angle between the first electronic device and the user's face.

18. An attention detection method based on facial orientation, characterized in that, The method is applied to a communication system, the communication system including a first electronic device and a second electronic device, the method comprising: The second electronic device responds to input events from the input device; The first electronic device determines a first angle between itself and the user's face, and the second electronic device determines a second angle between itself and the user's face; If the first electronic device determines that the first included angle is less than the second included angle, the first electronic device responds to the input event of the input device; The first electronic device responds to a first operation triggered by the user on the input device by executing the input event corresponding to the first operation.

19. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-16, or the method as described in claim 17, or the method performed by the first electronic device in the method as described in claim 18, or the method performed by the second electronic device in the method as described in claim 18.

20. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-16, or to perform the method as described in claim 17, or to perform the method performed by a first electronic device in the method as described in claim 18, or to perform the method performed by a second electronic device in the method as described in claim 18.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-16, or the method as described in claim 17, or the method as described in claim 18 performed by a first electronic device, or the method as described in claim 18 performed by a second electronic device.

22. A computer program product, characterized in that, The computer program product includes computer program code that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-16, or the method as described in claim 17, or the method performed by the first electronic device in the method as described in claim 18, or the method performed by the second electronic device in the method as described in claim 18.