Method and apparatus for displaying interface of smart wearable device, wearable device, and storage medium

Abstract

The present application provides a method and apparatus for displaying an interface of a smart wearable device, a wearable device, and a computer-readable storage medium. The method comprises: when it is detected that a wearer of a smart wearable device has target displacement data meeting an interaction condition, performing human body part detection; and when a target human body part is detected, displaying on the target human body part an interactive interface matching the target human body part, a body part category of the target human body part being the same as the body part category of an intended interaction body part of the target displacement data.

Description

Smart wearable device interface display method, device, wearable device and storage medium

[0001] This disclosure claims priority to Chinese Patent Application No. 202411967160.3, filed with the Chinese Patent Office on December 27, 2024, entitled "Method, Apparatus, Smart Wearable Device Interface Display, and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of interface display technology, specifically to a method, apparatus, smart wearable device, and computer-readable storage medium for displaying an interface on a smart wearable device. Background Technology

[0003] With the development of smart wearable technology, augmented reality (AR) and virtual reality (VR) devices have entered users' lives. Interface display is a core technology of AR and VR devices. Currently, the interface display of smart wearable devices such as AR and VR devices mainly projects the interactive interface according to pre-set projection parameters. However, the inventors of this application discovered during actual research and development that current device display solutions have a low degree of matching between the interface display format and the user's intended display format. For example, when a user uses the navigation interface provided by smart glasses while walking, if the user wants to look down at their watch, the smart glasses still display the interactive interface in the original projection position, failing to adaptively (e.g., according to arm position and size) move the interactive interface to the arm for display. Summary of the Invention

[0004] This application provides a method, apparatus, smart wearable device, and computer-readable storage medium for displaying an interface on a smart wearable device, which can improve the matching degree between the interface display form and the user's intention display form, so that the interactive interface display conforms to the user's habitual behavior or the user's general cognitive behavior.

[0005] In a first aspect, this application provides a method for displaying an interface on a smart wearable device, the method comprising:

[0006] Detecting displacement data of the wearer of a smart wearable device;

[0007] When target displacement data that meets the interaction conditions is detected in the wearer, the camera of the smart wearable device is invoked to detect human body parts.

[0008] When a target human body part is detected, an interactive interface matching the target human body part is displayed at the target human body part, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

[0009] Secondly, this application provides a smart wearable device interface display device, the smart wearable device interface display device comprising:

[0010] The first detection module is used to detect the displacement data of the wearer of the smart wearable device;

[0011] The second detection module is used to call the camera of the smart wearable device to detect human body parts when it detects that the wearer has target displacement data that meets the interaction conditions.

[0012] The display module is used to display an interactive interface matching the target human body part when a target human body part is detected, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

[0013] Thirdly, this application also provides a smart wearable device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes any of the smart wearable device interface display methods provided in this application when it calls the computer program in the memory.

[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the aforementioned smart wearable device interface display method.

[0015] In this application, firstly, when a target displacement data matching the interaction conditions of the wearer of the smart wearable device is detected, the camera is invoked to detect human body parts, and an interactive interface matching the target human body part is displayed on the target human body part. Since the target human body part displayed on the interactive interface is of the same category as the intentional interactive part of the target displacement data, and the intentional interactive part of the target displacement data can reflect the human body part that the user wants to display, the display position of the interactive interface matches the display position of the user's intention (e.g., when the target displacement data of looking down at a watch is detected, the interactive interface is displayed on the arm), thus improving the matching degree between the interface display form and the user's intention display form. Secondly, by displaying an interactive interface matching the target human body part on the target human body part, different interface content or interface layout forms can be set for different human body parts according to business needs, thus enabling the adaptive display of different content or layout forms of interactive interfaces for different human body parts, which can also improve the matching degree between the interface display form and the user's intention display form to a certain extent. Thirdly, since the corresponding interactive interface is triggered only when the wearer of the smart wearable device detects target displacement data that meets the interaction conditions, and the intended interactive part of the target displacement data can reflect the human body part that the user wants to display, the interactive interface can be displayed for user's habitual or generally recognized behaviors (such as displaying the interactive interface on the arm when the target displacement data of looking down at the watch is detected), so that the interactive interface displays behaviors that conform to the user's habitual or generally recognized behaviors, thereby improving the user experience of the interactive interface display. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a schematic block diagram of a smart wearable device provided in an embodiment of this application;

[0018] Figure 2 is a flowchart illustrating a smart wearable device interface display method provided in an embodiment of this application.

[0019] Figure 3 is a schematic diagram of a scenario in which an interactive interface is displayed on a target human body part according to an embodiment of this application.

[0020] Figure 4 is a schematic diagram of a user gesture action provided in an embodiment of this application;

[0021] Figure 5 is a schematic diagram of the changes in the interactive interface in response to user gestures provided in the embodiments of this application;

[0022] Figure 6 is a schematic flowchart of an embodiment of the smart wearable device interface display process provided in this application;

[0023] Figure 7 is a schematic diagram of an embodiment of the interface display device for a smart wearable device provided in this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0026] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0027] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0028] This application provides a method, apparatus, smart wearable device, and computer-readable storage medium for displaying an interface on a smart wearable device. The interface display apparatus can be integrated into the smart wearable device. The smart wearable device can be smart glasses, a smart helmet, etc. Smart glasses can be AR (augmented reality) glasses, VR (virtual reality) glasses, MR (mixed reality) glasses, XR (eXtended reality) glasses, etc., and smart helmets can be AR helmets, VR helmets, MR helmets, XR helmets, etc.

[0029] The execution subject of the smart wearable device interface display method in this application embodiment can be the smart wearable device interface display device provided in this application embodiment, or a smart wearable device that integrates the smart wearable device interface display device. The smart wearable device interface display device can be implemented in hardware or software.

[0030] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] Figure 1 is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of this application.

[0032] As shown in Figure 1, the smart wearable device 100 includes a processor 101 and a memory 102, which are connected by a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0033] Specifically, processor 101 provides computing and control capabilities to support the operation of the entire smart wearable device 100. Processor 101 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0034] Specifically, the memory 102 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0035] Those skilled in the art will understand that the structure shown in FIG1 is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the smart wearable device to which the embodiments of this application are applied. A specific smart wearable device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0036] The processor 101 is configured to run a computer program stored in the memory 102, and implement any of the smart wearable device interface display methods provided in this application embodiment when executing the computer program. For example, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps:

[0037] The system detects the displacement data of the wearer of the smart wearable device; when the wearer is detected to have target displacement data that meets the interaction conditions, the camera of the smart wearable device is invoked to detect human body parts; when a target human body part is detected, an interactive interface matching the target human body part is displayed on the target human body part, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

[0038] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the smart wearable device described above can be referred to the corresponding process in the following embodiments of the smart wearable device interface display method, and will not be repeated here.

[0039] The following will take the smart wearable device shown in Figure 1 as the execution subject of the smart wearable device interface display method as an example to describe in detail the smart wearable device interface display method provided in this application embodiment. For the sake of simplification and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0040] Please refer to Figure 2, which is a flowchart illustrating a method for displaying the interface of a smart wearable device according to an embodiment of this application. This method includes steps 201 to 203, wherein:

[0041] 201. Detect the displacement data of the wearer of the smart wearable device.

[0042] There are multiple ways to implement step 201, including, for example:

[0043] (1) In some embodiments, the displacement data includes head movement angle and head movement speed. The wearer's head movement angle and head movement speed can be detected sequentially to determine whether the wearer has target displacement data that meets the interaction conditions, so as to decide whether to trigger step 202. In this case, step 201 may specifically include the following steps 2011A to 2013A:

[0044] 2011A. Determine the head movement angle of the wearer based on the angle measurement data of the smart wearable device.

[0045] An inertial measurement unit (IMU) is a sensor system used to measure the acceleration and angular velocity of an object. In this embodiment, the smart wearable device is equipped with an inertial measurement unit to detect sensor data such as acceleration and angular velocity of the smart wearable device, thereby determining the head movement speed and angle of the wearer.

[0046] In some embodiments, the angle measurement data includes the angular velocity of the smart wearable device. The velocity measurement data can be obtained through the inertial measurement unit of the smart wearable device. The inertial measurement unit can be equipped with a gyroscope to measure the angular velocity of the smart wearable device (i.e., the rotational rate of the smart wearable device around a certain axis of the reference coordinate system, such as the x-axis; it can be understood that when the wearer wears the smart wearable device, the angular velocity of the smart wearable device is the angular velocity of the wearer's head, thereby providing information on the change of the angular velocity of the head around a certain direction). The head movement angle of the wearer can be determined based on the angular velocity of the smart wearable device and the relative positional relationship between the reference coordinate system of the smart wearable device and the wearer's head.

[0047] 2012A. When the head movement angle meets the preset angle conditions, the head movement speed of the wearer is determined according to the speed measurement data of the smart wearable device.

[0048] In some embodiments, the velocity measurement data includes the angular velocity and acceleration of the smart wearable device. This velocity measurement data can be acquired through the inertial measurement unit (IMU) of the smart wearable device. The IMU may be equipped with a gyroscope to measure the angular velocity of the smart wearable device (i.e., the rotational rate of the smart wearable device about a certain axis of the reference coordinate system, such as the x-axis). The IMU may also be equipped with an accelerometer to measure the acceleration of the smart wearable device (i.e., the movement speed of the smart wearable device in the direction of a certain axis of the reference coordinate system, such as the x-axis; understandably, when a wearer wears the smart wearable device, the acceleration of the smart wearable device is the acceleration of the wearer's head, thus providing information on the change in head acceleration in a certain direction). The wearer's head movement speed can be determined based on the angular velocity of the smart wearable device, the acceleration of the smart wearable device, and the relative positional relationship between the smart wearable device's reference coordinate system and the wearer's head.

[0049] Depending on the different settings of the preset angle conditions, step 2012A can be implemented in several ways, including, for example, the following methods ①, ② and ③:

[0050] ① In some embodiments, the head movement angle includes the vertical movement angle of the wearer's head relative to the horizontal plane, and the preset angle condition is that the vertical movement angle is within a first preset angle range. In this case, step 2012A may specifically include: when the vertical movement angle of the wearer's head relative to the horizontal plane is within the first preset angle range, determining the wearer's head movement speed based on the speed measurement data of the smart wearable device.

[0051] The specific value of the first preset angle range can be set according to the actual business scenario requirements. For example, when users look at a watch, their heads are generally tilted down about 15° to 20° relative to the horizontal plane. Therefore, when designing the interactive interface display for the user looking at a watch scenario, the first preset angle range can be set to [15°, 20°].

[0052] For example, in a scenario where it's necessary to determine whether a user is checking their watch to decide whether to display the interactive interface, it's necessary to determine if the wearer has target displacement data for checking the watch. In step 2011A, the angular velocity of the smart wearable device can be determined based on the angle measurement data from its inertial measurement unit. Furthermore, the vertical movement angle of the wearer's head relative to the horizontal plane can be determined based on the angular velocity of the smart wearable device, the relative position between the smart wearable device's reference coordinate system and the wearer's head. When step 2011A detects that the vertical movement angle is within a first preset angle range (e.g., [15°, 20°]), step 2012A is triggered to detect the wearer's head movement speed. Otherwise, if step 2011A detects that the vertical movement angle is outside the first preset angle range, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0053] ② In some embodiments, the head movement angle includes the left-right translation angle of the wearer's head relative to the horizontal plane, with the preset angle condition being that the left-right translation angle is within a second preset angle range. In this case, step 2012A may specifically include: when the left-right translation angle of the wearer's head relative to the horizontal plane is within the second preset angle range, determining the wearer's head movement speed based on the speed measurement data of the smart wearable device.

[0054] The specific value of the second preset angle range can be set according to the actual business scenario requirements. For example, when users look at a watch, their heads are generally shifted about 10° to 30° to the left or right relative to the horizontal. Therefore, when designing the interactive interface display for the user looking at a watch scenario, the second preset angle range can be set to [10°, 30°].

[0055] For example, in a scenario where it's necessary to determine whether a user is checking their watch to decide whether to display the interactive interface, it's necessary to determine if the wearer has target displacement data for checking the watch. In step 2011A, the angular velocity of the smart wearable device can be determined based on the angle measurement data from its inertial measurement unit. Furthermore, the left-right translation angle of the wearer's head relative to the horizontal plane can be determined based on the angular velocity of the smart wearable device, the relative position between the smart wearable device's reference coordinate system and the wearer's head. When step 2011A detects that the left-right translation angle is within a second preset angle range (e.g., [10°, 30°]), step 2012A is triggered to detect the wearer's head movement speed. Otherwise, if step 2011A detects that the left-right translation angle is outside the second preset angle range, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0056] ③ In some embodiments, the head movement angle includes the vertical movement angle and the horizontal translation angle of the wearer's head relative to the horizontal plane. The preset angle conditions are that the vertical movement angle is within a first preset angle range and the horizontal translation angle is within a second preset angle range. In this case, step 2012A may specifically include: when the vertical movement angle of the wearer's head relative to the horizontal plane is within the first preset angle range and the horizontal translation angle of the wearer's head relative to the horizontal plane is within the second preset angle range, the wearer's head movement speed is determined based on the speed measurement data of the smart wearable device. For example, in the scenario of "it is necessary to determine whether the user has performed a watch-checking action to decide whether to display the interactive interface", it is necessary to determine whether the wearer has target displacement data for the watch-checking action. In step 2011A, the angular velocity of the smart wearable device can be determined based on the angle measurement data of the inertial measurement unit of the smart wearable device. The vertical movement angle of the wearer's head relative to the horizontal plane and the horizontal translation angle of the wearer's head relative to the horizontal plane can be determined based on the angular velocity of the smart wearable device, the relative positional relationship between the smart wearable device's reference coordinate system and the wearer's head. When step 2011A detects that the vertical movement angle is within a first preset angle range (e.g., [15°, 20°]) and the horizontal translation angle is within a second preset angle range (e.g., [10°, 30°]), step 2012A is triggered to detect the wearer's head movement speed. Otherwise, when step 2011A detects that the vertical movement angle is outside the first preset angle range or the horizontal translation angle is outside the second preset angle range, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0057] 2013A. When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction conditions.

[0058] The first preset speed condition can be set according to the actual business scenario requirements, such as the head movement speed being within the first preset speed range.

[0059] The specific value of the first preset speed range can be set according to the actual business scenario requirements.

[0060] The target displacement data refers to the displacement data that meets the interaction conditions. The interaction conditions can be set according to the actual scenario requirements. There are no specific restrictions on the interaction conditions here. For example, in the implementation of steps 2011A to 2013A, the interaction conditions can be: the head movement angle meets the preset angle condition and the head movement speed meets the first preset speed condition.

[0061] For example, in a scenario where it's necessary to determine whether a user has checked their watch to decide whether to display the interactive interface, it's necessary to determine if the wearer has target displacement data indicating a watch-checking action. In step 2012A, the spatial position of the smart wearable device at various points in time can be determined based on its positioning data. Then, the wearer's head movement speed can be determined based on the smart wearable device's angular velocity, acceleration, and the relative position between the smart wearable device's reference coordinate system and the wearer's head. If the head movement speed is detected to be within a first preset speed range in step 2012A, step 2013 is triggered, determining that the wearer has target displacement data that meets the interaction conditions. Otherwise, if the head movement speed is detected to be outside the first preset speed range in step 2012A, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0062] (2) In some embodiments, the displacement data includes overall movement speed, head movement angle, and head movement speed. The overall movement speed, head movement angle, and head movement speed of the wearer can be detected sequentially to determine whether the wearer has target displacement data that meets the interaction conditions, so as to decide whether to trigger step 202. In this case, step 201 may specifically include the following steps 2011B to 2014B:

[0063] 2011B. Determine the wearer's overall movement speed based on the positioning data of the smart wearable device.

[0064] In some embodiments, the positioning data of the smart wearable device can be obtained through its positioning unit. This positioning unit can be GPS (Global Positioning System), a system that uses satellite signals for positioning, navigation, and timing, providing accurate geographic location data. By collecting positioning data from the smart wearable device's positioning unit (such as GPS), the spatial location of the smart wearable device at various points in time can be determined, allowing the wearer's overall movement speed to be calculated based on the device's spatial location at each point in time.

[0065] Since users may pause or slow down their movement when viewing or performing certain interactive interface operations, such as when viewing a watch, this embodiment of the application uses positioning data to determine the wearer's overall movement speed in order to improve the accuracy of determining the viewing or interaction intention of the interactive interface.

[0066] 2012B. When the overall movement speed meets the second preset speed condition, the head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device.

[0067] The second preset speed condition can be set according to the actual business scenario requirements, such as the overall movement speed being within the second preset speed range.

[0068] The specific value of the second preset speed range can be set according to the actual business scenario requirements. For example, it can be set to less than or equal to 0.01m / s.

[0069] For example, in a scenario where it's necessary to determine whether a user has checked their watch to decide whether to display the interactive interface, it's necessary to determine if the wearer has target displacement data for checking the watch. In step 2011B, the spatial position of the smart wearable device at each time point can be determined based on its positioning data. Then, the wearer's overall movement speed can be calculated based on the smart wearable device's spatial position at each time point. When step 2011B detects that the overall movement speed is within a second preset speed range (e.g., less than or equal to 0.01 m / s), step 2012B is triggered to detect the wearer's head movement angle. Otherwise, when step 2012B detects that the overall movement speed is outside the second preset speed range, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0070] 2013B. When the head movement angle meets the preset angle conditions, the head movement speed of the wearer is determined based on the speed measurement data of the smart wearable device.

[0071] 2014B. When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction conditions.

[0072] (3) In some embodiments, the wearer's overall movement speed, head movement angle, and head movement speed can be detected in parallel to determine whether the wearer has target displacement data that meets the interaction conditions, so as to decide whether to trigger step 202. In this case, step 201 may specifically include the following steps 2011C to 2014C:

[0073] 2011C. Determine the wearer's overall movement speed based on the positioning data of the smart wearable device.

[0074] The implementation of step 2011C is similar to that of step 2011B. For details, please refer to the relevant explanations above. It will not be repeated here.

[0075] 2012C. Determine the head movement angle of the wearer based on the angle measurement data of the smart wearable device.

[0076] The implementation of step 2012C is similar to that of step 2011A. For details, please refer to the relevant explanations above. It will not be repeated here.

[0077] 2013C. Determine the head movement speed of the wearer based on the speed measurement data of the smart wearable device.

[0078] The implementation of step 2013C is similar to that of step 2012A. For details, please refer to the relevant explanations above. It will not be repeated here.

[0079] 2014C. When the overall movement speed meets the second preset speed condition, the head movement angle meets the preset angle condition, and the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction conditions.

[0080] For the specific implementation of steps 2011C to 2014C, please refer to the relevant descriptions of steps 2011A to 2013A and steps 2011B to 2014B, which will not be repeated here.

[0081] 202. When the wearer has target displacement data that meets the interaction conditions, the camera of the smart wearable device is invoked to detect human body parts.

[0082] Among them, smart wearable devices can be smart glasses, smart helmets, etc.; smart glasses can be AR glasses, VR glasses, MR glasses, XR glasses, etc.; and smart helmets can be AR helmets, etc.

[0083] In some embodiments, when the wearer has target displacement data that meets the interaction conditions, the camera of the smart wearable device can be invoked to capture scene images; human body parts are detected based on the scene images using a trained part detection model, and the human body part detection results of the scene images are obtained, wherein the human body part detection results include the location information and part category of the target human body parts.

[0084] Among them, key points of the human body refer to key points of various parts of the human body, such as key points of the head, eyes, shoulders, elbows, wrists, knees, and ankles.

[0085] For example, human region detection using a trained part detection model includes the following steps:

[0086] 1. Key point identification: Set one or more body key points according to the needs of the scene. For example, the key points to be set include: head key points, eye key points, shoulder key points, elbow key points, wrist key points, knee key points, ankle key points, etc.

[0087] 2. Collect sample images of human body key points (e.g., wrist key points) for each body part category i, and train a pre-defined body part detection model. This trained model can identify whether each pixel in the image is a human body key point of body part category i (e.g., whether it is a wrist key point), and based on the pixels in the image that belong to body body key points of body part category i (e.g., pixels belonging to wrist key points), it can segment the human body part of body part category i from the image (e.g., segmenting the wrist part based on pixels identified as "belonging to wrist key points"). For example, a body part detection model can be constructed using convolutional neural networks (CNN), graph convolutional networks (GCN), etc., to predict whether each pixel in the image belongs to human body key points of body part category i, thereby segmenting the human body part of body part category i.

[0088] 3. In step 203, when target displacement data matching the interaction conditions is detected in the wearer of the smart wearable device, the camera of the smart wearable device is invoked to capture scene images. Using a trained part detection model, keypoint detection for part category i is performed based on the scene images, resulting in a pixel classification map i indicating whether each pixel in the scene image belongs to a human keypoint of part category i (where the pixel value of each pixel in the pixel classification map i indicates the location of the pixel belonging to the human keypoint of part category i). Then, based on the pixel value of each pixel in the pixel classification map i, it is determined whether the corresponding pixel in the scene image is the pixel of the human keypoint of part category i, thus determining the pixels of all human keypoints of part category i (such as wrist keypoints) in the scene image. The scene image is then divided according to the pixels of all human keypoints of part category i in the scene image to obtain the human parts of part category i. Similarly, human parts of all part categories in the scene image can be detected, such as the head, eyes, shoulders, elbows, wrists, knees, and ankles.

[0089] Furthermore, based on the human body part detection results (e.g., based on the part categories of human body parts contained in the scene image), it can be determined whether there is a target human body part in the scene image that has the same part category as the intended interaction part. If the scene image contains a human body part with the same part category as the intended interaction part, it is determined that a target human body part has been detected, and the human body part in the scene image with the same part category as the intended interaction part is taken as the target human body part; if the scene image does not contain a human body part with the same part category as the intended interaction part, it is determined that no target human body part has been detected. For example, after detecting human body parts of all part categories in the scene image, it is also compared whether there is a human body part with the same part category as the intended interaction part. If it exists, it is determined that a target human body part has been detected, and the human body part with the same part category as the intended interaction part is taken as the target human body part; if it does not exist, it is determined that no target human body part has been detected. Among them, the intentional interaction part is the human body part that the user intends to interact with, corresponding to the target displacement data. For example, if the human body part that the user intends to interact with is the left arm, then the target displacement data will be "the overall movement speed is 0.1m / s, the head moves downward at an angle of 15° and to the left at an angle of 10°, and the head movement speed is within the first preset speed range".

[0090] In this way, the camera of the smart wearable device can be used to capture scene images, and the scene images can be identified to determine whether there are human body parts within the field of view of the camera. When there are human body parts in the scene image, the scene image can be further detected by the trained part detection model to determine the location information of the target human body parts in the scene image and the part category of the target human body parts.

[0091] Furthermore, to prevent overheating and power consumption issues caused by prolonged camera operation, the camera is turned off if no human body is detected for more than a preset time threshold after being activated. The specific value of the preset time threshold can be set according to actual business needs. For example, in a scenario where it's necessary to determine whether a user is checking their watch to decide whether to display an interactive interface, an average person typically spends 2 to 5 seconds checking the time. In emergencies or quick checks, it might only take 1 to 2 seconds, while additional functions or more information could require 5 to 10 seconds. Therefore, a human body (such as an arm) will generally appear within the camera's field of view within 10 seconds, so the preset time threshold can be set to 10 seconds.

[0092] 203. When a target human body part is detected, an interactive interface matching the target human body part is displayed on the target human body part.

[0093] The part category of the target human body part is the same as the part category of the intended interaction part of the target displacement data.

[0094] When a target human body part is detected, an interface projection area is determined from the target human body part based on its location information; an interactive interface matching the target human body part is then displayed in the interface projection area. In some embodiments, the interface projection area can be determined from the target human body part according to preset layout information and the location information of the target human body part. In some embodiments, the interface projection area can be determined from the target human body part according to interface layout information matching the part category of the target human body part and the location information of the target human body part.

[0095] There are several ways to implement step 203, including, for example:

[0096] (1) In some embodiments, each body part category corresponds to a set of interface content to adapt to different human body parts displaying different content interactive interfaces, thereby improving the richness of the interface display interaction. In this case, step 203 may include step 2031A:

[0097] 2031A. When a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface content matched with the part category of the target human body part.

[0098] For example, as shown in Table 1, assuming that there is a preset mapping relationship between each body part category and the interface content as shown in Table 1, when a target human body part (such as the arm, i.e., the body part category of the target human body part is arm) is detected, an interactive interface containing the body part category matching the target human body part (i.e., displaying content 1) is displayed at the target human body part, as shown in Figure 3(a). In this way, different human body parts can be matched with interactive interfaces containing different interface content.

[0099] It is understandable that the interaction intent can be further identified based on the target displacement data; based on the interaction intent and the part category of the target human body, the interface content that matches the part category of the target human body and the interaction intent can be determined, as shown in Figure 3(a) and (b). In this way, the same human body part can also be matched with an interactive interface with different interface content, thereby improving the richness of the interaction.

[0100] Table 1

[0101] (2) In some embodiments, each body part category corresponds to an interface layout information to adapt to different sizes of interactive interfaces displayed for different human body parts, thereby improving the fit between the interactive interface and the human body parts. In this case, step 203 may include step 2031B:

[0102] 2031B. When a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface layout information matched with the part category of the target human body part, wherein the interface layout information includes at least one of interface ratio and interface size.

[0103] For example, when a target human body part is detected, an interface projection area is determined from the target human body part based on the location information of the target human body part and the interface layout information matching the part category of the target human body part; projection is performed on the interface projection area to display an interactive interface matching the target human body part on the target human body part.

[0104] For example, as shown in Table 2, assuming that there is a preset mapping relationship between each body part category and the interface layout information as shown in Table 2, when a target body part (such as an arm, i.e., the target body part category is arm) is detected, the interactive interface displayed on the target body part will be based on the interface layout information that matches the target body part category (i.e., according to interface layout a). In this way, different body parts can be matched with interactive interfaces of different interface layouts.

[0105] Table 2

[0106] (3) In some embodiments, each part category corresponds to a type of interface content and each part category corresponds to a type of interface layout information. In this case, step 203 may include step 2031C:

[0107] 2031C. When target displacement data matching the interaction conditions is detected in the wearer of the smart wearable device, an interactive interface is displayed on the target human body part according to the interface content matching the part category and the interface layout information matching the part category.

[0108] For example, when a target human body part is detected, an interface projection area is determined from the target human body part based on the location information of the target human body part and the interface layout information that matches the part category of the target human body part; the interface projection area is projected to display an interactive interface containing interface content that matches the part category of the target human body part, so as to realize the display of an interactive interface that matches the target human body part on the target human body part.

[0109] Furthermore, the smart wearable device interface display method also includes: when a user gesture is detected, updating the display content of the interactive interface according to the user gesture. As shown in Figure 4, which is a schematic diagram of a user gesture provided in this embodiment, after the interactive interface is displayed on the target human body part, when the camera recognizes the user gesture during the opening process, such as clicking or double-clicking, the smart wearable device can respond to the user gesture and update the display content of the interactive interface according to the corresponding user gesture. For example, as shown in Figure 5, which is a schematic diagram of the change of the interactive interface in response to the user gesture in this embodiment, Figure 5(a) is the SMS preview page and Figure 5(b) is the SMS details page. After the interactive interface shown in Figure 5(a) is displayed on the arm in step 203, if the user clicks the SMS preview page, it will jump to the SMS details page shown in Figure 5(b), thereby realizing the updating of the display content of the interactive interface through the user gesture.

[0110] For ease of understanding, please refer to Figure 6. The following example, using "an inertial measurement unit (IMU) and a positioning unit (GPS)," illustrates the interface display process of this smart wearable device:

[0111] S1. Detect whether the wearer has target displacement data that meets the interaction conditions using IMU and GPS. (Refer to the "Detection of target displacement data that meets the interaction conditions" section in Figure 6)

[0112] At this point, the IMU and GPS can determine whether preliminary step S2 is needed to activate the camera to match the user's movements based on three conditions:

[0113] The first condition: The user's overall movement speed currently meets the second preset speed condition. When the user needs to check their watch, they will generally pause or slow down their movement. At this time, GPS can be used to determine whether the user's overall movement speed meets the second preset speed condition. For example, if the user's overall movement speed is within the second preset speed range and approaches a certain value (such as less than or equal to 0.01 m / s), it is considered that the overall movement speed meets the second preset speed condition, triggering the judgment of the second condition.

[0114] The second condition is that the user's head movement angle meets the preset angle conditions. At this time, the IMU can be used to determine whether the user's head movement angle meets the preset angle conditions. For example, if the vertical movement angle is within the first preset angle range (e.g., [15°, 20°]) and the horizontal movement angle is within the second preset angle range (e.g., [10°, 30°]), the head movement angle is considered to meet the preset angle conditions, triggering the judgment of the third condition.

[0115] The third condition: The user's head movement speed meets the first preset speed condition. At this time, the IMU can be used to determine whether the user's head movement speed meets the first preset speed condition. For example, if the head movement speed is within the first preset speed range, it is considered that the user's head movement speed meets the first preset speed condition, and step S2 is triggered.

[0116] S2. Turn on the camera to preview. (Refer to the "Camera On" section in Figure 6)

[0117] To prevent overheating and power consumption issues caused by prolonged camera operation, the camera will be turned off if no human body is detected within a preset time threshold after the camera has been on for an extended period.

[0118] S3. Target human body part (e.g., arm) detected. Detailed implementation can be found in the previous explanations and will not be repeated here. (Refer to the "Whether human body part was detected" section in Figure 6)

[0119] S4. Project the relevant interactive interface onto the target body part (such as the arm). (Refer to the "Device Displays Relevant Interactive Interface" section in Figure 6)

[0120] Taking the recognition of an arm as an example, the corresponding interactive interface is displayed in the arm area based on the obtained arm position information.

[0121] S5. In response to the user's gesture, the interactive interface displayed on the target body part (such as the arm) changes. (Refer to the "Gesture Detection?" and "Device Responds to Gesture" sections in Figure 6)

[0122] As can be seen from the above, firstly, by detecting target displacement data that meets the interaction conditions of the wearer of the smart wearable device, the camera is invoked to detect human body parts, and an interactive interface matching the target human body part is displayed on the target human body part. Since the target human body part displayed on the interactive interface is of the same category as the intended interactive part of the target displacement data, and the intended interactive part of the target displacement data can reflect the human body part that the user wants to display, the display position of the interactive interface matches the display position of the user's intention (e.g., when the target displacement data of looking down at a watch is detected, the interactive interface is displayed on the arm). This can improve the matching degree between the interface display form and the user's intention display form. Secondly, by displaying an interactive interface that matches the target human body part on the target human body part, different interface content or interface layout forms can be set for different human body parts according to business needs. This allows for the adaptive display of different content or different layout forms of interactive interfaces for different human body parts, which can also improve the matching degree between the interface display form and the user's intention display form to a certain extent. Thirdly, since the corresponding interactive interface is triggered only when the wearer of the smart wearable device detects target displacement data that meets the interaction conditions, and the intended interactive part of the target displacement data can reflect the human body part that the user wants to display, the interactive interface can be displayed for user's habitual or generally recognized behaviors (such as displaying the interactive interface on the arm when the target displacement data of looking down at the watch is detected), so that the interactive interface displays behaviors that conform to the user's habitual or generally recognized behaviors, thereby improving the user experience of the interactive interface display.

[0123] Furthermore, to better implement the smart wearable device interface display method in this application embodiment, based on the smart wearable device interface display method, this application embodiment also provides a smart wearable device interface display device, as shown in FIG7, which is a schematic diagram of an embodiment of the smart wearable device interface display device provided in this application embodiment. The smart wearable device interface display device 700 includes:

[0124] First detection module 701, the first detection module, is used to detect the displacement data of the wearer of the smart wearable device;

[0125] The second detection module 702 is used to call the camera of the smart wearable device to detect human body parts when it detects that the wearer has target displacement data that meets the interaction conditions.

[0126] The display module 703 is used to display an interactive interface matching the target human body part when a target human body part is detected, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

[0127] In some embodiments, the display module 703 is specifically used for:

[0128] When a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface content matched with the part category of the target human body part.

[0129] And / or, when a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface layout information matched with the part category of the target human body part, wherein the interface layout information includes at least one of interface ratio and interface size.

[0130] In some embodiments, the second detection module 702 is specifically used for:

[0131] When the wearer is detected to have target displacement data that meets the interaction conditions, the camera of the smart wearable device is invoked to capture scene images.

[0132] By using a trained part detection model, human body parts are detected based on the scene image to obtain the human body part detection results of the scene image. The human body part detection results include the location information and part category of each human body part contained in the scene image.

[0133] In some embodiments, the second detection module 702 is specifically used for:

[0134] If the scene image contains a human body part of the same category as the part of the intentional interaction area, then it is determined that a target human body part has been detected, and the human body part of the scene image that is of the same category as the part of the intentional interaction area is taken as the target human body part.

[0135] In some embodiments, the display module 703 is specifically used for:

[0136] When a target human body part is detected, the interface projection area is determined from the target human body part based on the location information of the target human body part;

[0137] An interactive interface matching the target human body part is displayed in the interface projection area.

[0138] In some embodiments, the displacement data includes head movement angle and head movement speed, and the first detection module 701 is specifically used for:

[0139] The head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device;

[0140] When the head movement angle meets the preset angle conditions, the head movement speed of the wearer is determined based on the speed measurement data of the smart wearable device;

[0141] When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction conditions.

[0142] In some embodiments, the displacement data further includes the overall moving speed, and the display module 703 is specifically used for:

[0143] The wearer's overall movement speed is determined based on the positioning data from the smart wearable device.

[0144] Determining the wearer's head movement angle based on the angle measurement data from the smart wearable device includes:

[0145] When the overall movement speed meets the second preset speed condition, the head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device.

[0146] In some embodiments, the display module 703 is specifically used for:

[0147] When a user gesture is detected, the content displayed on the interactive interface is updated according to the user gesture.

[0148] In some embodiments, the second detection module 702 is specifically used for:

[0149] If no human body part is detected after the camera of the smart wearable device is invoked for a preset time threshold, the camera is turned off.

[0150] In practice, each of the above units can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous embodiment of the intelligent wearable device interface display method, which will not be repeated here.

[0151] Those skilled in the art will understand that all or part of the steps in the above-described intelligent wearable device interface display method can be completed by instructions, or by controlling related hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0152] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute any of the smart wearable device interface display methods provided in embodiments of this application.

[0153] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0154] In the above embodiments of the smart wearable device interface display device, computer-readable storage medium, and smart wearable device, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the smart wearable device interface display device, computer-readable storage medium, smart wearable device, and their corresponding units described above can be referred to the description of the smart wearable device interface display method in the above embodiments, and will not be repeated here.

[0155] The present application provides a detailed description of a smart wearable device interface display method, apparatus, smart wearable device, and computer-readable storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A method for displaying an interface on a smart wearable device, the method comprising: Detecting displacement data of the wearer of a smart wearable device; When target displacement data that meets the interaction conditions is detected in the wearer, the camera of the smart wearable device is invoked to detect human body parts. When a target human body part is detected, an interactive interface matching the target human body part is displayed at the target human body part, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

2. The smart wearable device interface display method according to claim 1, wherein, When a target human body part is detected, displaying an interactive interface matching the target human body part at that target human body part includes: When a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface content matched with the part category of the target human body part. And / or, when a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface layout information matched with the part category of the target human body part, wherein the interface layout information includes at least one of interface ratio and interface size.

3. The smart wearable device interface display method according to claim 1, wherein, When the wearer is detected to have target displacement data that meets the interaction conditions, the camera of the smart wearable device is invoked to detect human body parts, including: When the wearer is detected to have target displacement data that meets the interaction conditions, the camera of the smart wearable device is invoked to capture scene images. By using a trained part detection model, human body parts are detected based on the scene image to obtain the human body part detection results of the scene image. The human body part detection results include the location information and part category of each human body part contained in the scene image.

4. The smart wearable device interface display method according to claim 3, wherein, The method further includes: If the scene image contains a human body part of the same category as the part of the intentional interaction area, then it is determined that a target human body part has been detected, and the human body part of the scene image that is of the same category as the part of the intentional interaction area is taken as the target human body part. When a target human body part is detected, displaying an interactive interface matching the target human body part at that target human body part includes: When a target human body part is detected, the interface projection area is determined from the target human body part based on the location information of the target human body part; An interactive interface matching the target human body part is displayed in the interface projection area.

5. The smart wearable device interface display method according to claim 1, wherein, The displacement data includes the head movement angle and head movement speed. The displacement data of the wearer of the smart wearable device includes: The head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device; When the head movement angle meets the preset angle conditions, the head movement speed of the wearer is determined based on the speed measurement data of the smart wearable device; When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction conditions.

6. The smart wearable device interface display method according to claim 5, wherein, The displacement data also includes the overall movement speed, and the method further includes: The wearer's overall movement speed is determined based on the positioning data from the smart wearable device. Determining the wearer's head movement angle based on the angle measurement data from the smart wearable device includes: When the overall movement speed meets the second preset speed condition, the head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device.

7. The smart wearable device interface display method according to claim 5, wherein, The head movement angle includes the vertical movement angle and the horizontal translation angle of the wearer's head relative to the horizontal plane. The preset angle conditions are that the vertical movement angle is within a first preset angle range and the horizontal translation angle is within a second preset angle range. When the head movement angle meets the preset angle conditions, the wearer's head movement speed is determined based on the speed measurement data of the smart wearable device, including: When the vertical movement angle of the wearer's head relative to the horizontal plane is within a first preset angle range, and the horizontal movement angle of the wearer's head relative to the horizontal plane is within a second preset angle range, the head movement speed of the wearer is determined based on the speed measurement data of the smart wearable device.

8. The smart wearable device interface display method according to claim 1, wherein, The method further includes: When a user gesture is detected, the content displayed on the interactive interface is updated according to the user gesture.

9. The smart wearable device interface display method according to claim 1, wherein, The method further includes: If no human body part is detected after the camera of the smart wearable device is invoked for a preset time threshold, the camera is turned off.

10. A smart wearable device interface display device, the smart wearable device interface display device comprising: The first detection module is used to detect the displacement data of the wearer of the smart wearable device; The second detection module is used to call the camera of the smart wearable device to detect human body parts when it detects that the wearer has target displacement data that meets the interaction conditions. The display module is used to display an interactive interface matching the target human body part when a target human body part is detected, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

11. The smart wearable device interface display device according to claim 10, wherein, The display module is used for: When a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface content matched with the part category of the target human body part. And / or, when a target human body part is detected, an interactive interface is displayed on the target human body part according to the interface layout information matched with the part category of the target human body part, wherein the interface layout information includes at least one of interface ratio and interface size.

12. The smart wearable device interface display device according to claim 10, wherein, The second detection module is used for: When the wearer is detected to have target displacement data that meets the interaction conditions, the camera of the smart wearable device is invoked to capture scene images. By using a trained part detection model, human body parts are detected based on the scene image to obtain the human body part detection results of the scene image. The human body part detection results include the location information and part category of each human body part contained in the scene image.

13. The smart wearable device interface display device according to claim 12, wherein, The second detection module is used for: If the scene image contains a human body part of the same category as the part of the intentional interaction area, then it is determined that a target human body part has been detected, and the human body part of the scene image that is of the same category as the part of the intentional interaction area is taken as the target human body part. The display module is used for: When a target human body part is detected, the interface projection area is determined from the target human body part based on the location information of the target human body part; An interactive interface matching the target human body part is displayed in the interface projection area.

14. The smart wearable device interface display device according to claim 10, wherein, The displacement data includes the head movement angle and the head movement speed, and the first detection module is used for: The head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device; When the head movement angle meets the preset angle conditions, the head movement speed of the wearer is determined based on the speed measurement data of the smart wearable device; When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction conditions.

15. The smart wearable device interface display device according to claim 14, wherein, The displacement data also includes the overall movement speed, and the display module is used for: The wearer's overall movement speed is determined based on the positioning data from the smart wearable device. The first detection module is configured to include: When the overall movement speed meets the second preset speed condition, the head movement angle of the wearer is determined based on the angle measurement data of the smart wearable device.

16. The smart wearable device interface display device according to claim 14, wherein, The head movement angle includes the vertical movement angle and the horizontal translation angle of the wearer's head relative to the horizontal plane. The preset angle conditions are that the vertical movement angle is within a first preset angle range and the horizontal translation angle is within a second preset angle range. The first detection module is used for: When the vertical movement angle of the wearer's head relative to the horizontal plane is within a first preset angle range, and the horizontal movement angle of the wearer's head relative to the horizontal plane is within a second preset angle range, the head movement speed of the wearer is determined based on the speed measurement data of the smart wearable device.

17. The smart wearable device interface display device according to claim 10, wherein, The display module is used for: When a user gesture is detected, the content displayed on the interactive interface is updated according to the user gesture.

18. The smart wearable device interface display device according to claim 10, wherein, The second detection module is used for: If no human body part is detected after the camera of the smart wearable device is invoked for a preset time threshold, the camera is turned off.

19. A smart wearable device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the smart wearable device interface display method as described in any one of claims 1 to 9 when it invokes the computer program in the memory.

20. A computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the smart wearable device interface display method according to any one of claims 1 to 9.

Images