Control method and apparatus for auxiliary robot

Abstract

A control method for an auxiliary robot. The method comprises: acquiring a first image collected by means of a collection device, wherein the first image represents the distance between a contact component of a target subject and the collection device and the distance between a contact surface of an environment where the target subject (103) is located and the collection device, respectively; segmenting a first region and a second region from the first image, wherein the first region represents the distance between the contact component and the collection device, and the second region represents the distance between the contact surface and the collection device; on the basis of the first region and the second region, determining a first contact state of the target subject, wherein the first contact state indicates whether the contact component is in contact with the contact surface; and on the basis of the first contact state, controlling an auxiliary robot, wherein the auxiliary robot is used for assisting movements of the target subject. Further disclosed are a control apparatus for an auxiliary robot, and an electronic device, a computer-readable storage medium and a computer program product. An auxiliary robot is controlled on the basis of a first contact state, such that the auxiliary robot assists a target subject in completing movements such as walking and climbing.

Description

Control methods and devices for assisted robots

[0001] This application claims priority to Chinese Patent Application No. 202411560673.2, filed on November 4, 2024, entitled "Control Method and Apparatus for Assistive Robot", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of robotics, and in particular to a control method and apparatus for an auxiliary robot. Background Technology

[0003] With the development of robotics technology, robots are becoming increasingly versatile. Some robots possess assistive capabilities, allowing them to be attached to objects and assist in tasks such as walking and climbing. These robots with assistive capabilities are called assistive robots, and how to control them has become a pressing problem to be solved. Summary of the Invention

[0004] This application provides a control method and device for an auxiliary robot, which can be used to solve problems in related technologies. The technical solution includes the following contents.

[0005] Firstly, a control method for an auxiliary robot is provided, the method comprising:

[0006] A first image is acquired by the acquisition device, wherein the first image represents the distance between the contact parts of the target object, the contact surface of the environment in which the target object is located, and the acquisition device.

[0007] A first region and a second region are segmented from the first image. The first region represents the distance between the contact component and the acquisition device, and the second region represents the distance between the contact surface and the acquisition device.

[0008] Based on the first region and the second region, a first contact state of the target object is determined, wherein the first contact state characterizes whether the contact component contacts the contact surface;

[0009] The auxiliary robot is controlled according to the first contact state, and the auxiliary robot is used to assist the target object in moving.

[0010] In one possible implementation, segmenting the first region and the second region from the first image includes:

[0011] The first image is preprocessed to obtain a preprocessed image, wherein the preprocessing includes at least one of noise reduction, image cropping, and information filtering;

[0012] The first region and the second region are segmented from the preprocessed image.

[0013] In one possible implementation, segmenting the first region and the second region from the preprocessed image includes:

[0014] The preprocessed image is segmented using an image segmentation model to obtain a first mask image and a second mask image. The first mask image is used to describe whether each pixel in the preprocessed image belongs to the contact component, and the second mask image is used to describe whether each pixel belongs to the contact surface.

[0015] Based on the first mask image, determine the first region;

[0016] The second region is determined based on the second mask image.

[0017] In one possible implementation, determining the first contact state of the target object based on the first region and the second region includes:

[0018] The distance between the contact component and the contact surface is determined based on the first region and the second region;

[0019] If the distance is less than a threshold, the first contact state is determined to represent that the contact component is in contact with the contact surface;

[0020] If the distance is not less than the threshold, the first contact state is determined to indicate that the contact component is not in contact with the contact surface.

[0021] In one possible implementation, the method further includes:

[0022] Acquire a series of second images captured by the acquisition device, wherein the second images represent the distances between the contact component, the contact surface, and the acquisition device, respectively;

[0023] Multiple second contact states are determined based on the multiple frames of the second images;

[0024] First motion information of the target object is determined based on the plurality of second contact states, and the first motion information is used to characterize the posture and behavioral features of the target object during motion.

[0025] The step of controlling the auxiliary robot according to the first contact state includes:

[0026] The robot is controlled based on the first motion information and the first contact state.

[0027] In one possible implementation, the target object includes multiple supporting components and multiple movable components, the movable components being used to control the supporting components, and the method further includes:

[0028] Acquire first motion data of the plurality of supporting components and second motion data of the plurality of movable components;

[0029] Based on the first motion data and the second motion data, second motion information of the target object is determined, wherein the second motion information is used to characterize the posture and behavioral features of the target object during motion;

[0030] The step of controlling the auxiliary robot according to the first contact state includes:

[0031] The robot is controlled based on the second motion information and the first contact state.

[0032] In one possible implementation, determining the second motion information of the target object based on the first motion data and the second motion data includes:

[0033] The first motion data and the second motion data are preprocessed to obtain preprocessed data, wherein the preprocessing includes at least one of noise reduction processing and calibration processing.

[0034] Based on the preprocessed data, the second motion information of the target object is determined.

[0035] In one possible implementation, determining the second motion information of the target object based on the first motion data and the second motion data includes:

[0036] Based on the first motion data, determine the attitude information of the plurality of support components;

[0037] Based on the multiple posture information and the second motion data, multiple relative motion information is determined, and one relative motion information is used to describe the motion of one support component relative to another support component;

[0038] Based on the plurality of relative motion information, the second motion information of the target object is determined.

[0039] Secondly, a control device for an auxiliary robot is provided, the device comprising:

[0040] The acquisition module is used to acquire a first image captured by the acquisition device, wherein the first image represents the distance between the contact parts of the target object, the contact surface of the environment in which the target object is located, and the acquisition device.

[0041] A segmentation module is used to segment a first region and a second region from the first image, wherein the first region represents the distance between the contact component and the acquisition device, and the second region represents the distance between the contact surface and the acquisition device;

[0042] The determining module is configured to determine a first contact state of the target object based on the first region and the second region, wherein the first contact state characterizes whether the contact component contacts the contact surface;

[0043] A control module is used to control an auxiliary robot based on the first contact state, the auxiliary robot being used to assist the target object in its movement.

[0044] In one possible implementation, the segmentation module is used to preprocess the first image to obtain a preprocessed image, the preprocessing including at least one of denoising, image cropping, and information filtering; and to segment a first region and a second region from the preprocessed image.

[0045] In one possible implementation, the segmentation module is used to perform image segmentation on the preprocessed image using an image segmentation model to obtain a first mask image and a second mask image. The first mask image is used to describe whether each pixel in the preprocessed image belongs to the contact component, and the second mask image is used to describe whether each pixel belongs to the contact surface. A first region is determined based on the first mask image, and a second region is determined based on the second mask image.

[0046] In one possible implementation, the determining module is configured to determine the distance between the contact component and the contact surface based on the first region and the second region; if the distance is less than a threshold, determine the first contact state to indicate that the contact component is in contact with the contact surface; if the distance is not less than the threshold, determine the first contact state to indicate that the contact component is not in contact with the contact surface.

[0047] In one possible implementation, the acquisition module is further configured to acquire multiple consecutive frames of second images acquired by the acquisition device, wherein the second images represent the distances between the contact component, the contact surface, and the acquisition device, respectively.

[0048] The determining module is further configured to determine multiple second contact states based on the multiple frames of the second image; and to determine first motion information of the target object based on the multiple second contact states, wherein the first motion information is used to characterize the posture and behavioral features of the target object during motion;

[0049] The control module is used to control the auxiliary robot based on the first motion information and the first contact state.

[0050] In one possible implementation, the target object includes multiple supporting components and multiple movable components, the movable components being used to control the supporting components;

[0051] The acquisition module is further configured to acquire first motion data of the plurality of support components and second motion data of the plurality of movable components;

[0052] The determining module is further configured to determine second motion information of the target object based on the first motion data and the second motion data, wherein the second motion information is used to characterize the posture and behavioral features of the target object during motion;

[0053] The control module is used to control the auxiliary robot based on the second motion information and the first contact state.

[0054] In one possible implementation, the determining module is configured to preprocess the first motion data and the second motion data to obtain preprocessed data, wherein the preprocessing includes at least one of noise reduction processing and calibration processing; and determine the second motion information of the target object based on the preprocessed data.

[0055] In one possible implementation, the determining module is configured to determine the posture information of the plurality of support components based on the first motion data; determine a plurality of relative motion information based on the plurality of posture information and the second motion data, wherein one relative motion information describes the motion of one support component relative to another support component; and determine the second motion information of the target object based on the plurality of relative motion information.

[0056] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the above-described control methods for assistive robots.

[0057] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement any of the above-described control methods for the auxiliary robot.

[0058] Fifthly, a computer program is also provided, wherein the computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement any of the above-described control methods for assistive robots.

[0059] Sixthly, a computer program product is also provided, wherein at least one computer program is stored in the computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-described control methods for an auxiliary robot.

[0060] The technical solution provided in this application brings at least the following beneficial effects:

[0061] In the technical solution provided in this application, a first image is acquired using an acquisition device, and a first region and a second region are segmented from the first image. Since the first region represents the distance between the contacting component of the target object and the acquisition device, and the second region represents the distance between the contact surface of the target object's environment and the acquisition device, it is possible to determine whether the contacting component is in contact with the contact surface based on the first and second regions, thus obtaining the first contact state. During the movement of the target object, it will periodically alternate between contacting the contacting surface and not contacting it. Based on this, an auxiliary robot can be controlled according to the first contact state to assist the target object in completing movements such as walking and climbing. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 is a schematic diagram of a computer system provided in an embodiment of this application;

[0064] Figure 2 is a flowchart of a control method for an auxiliary robot provided in an embodiment of this application;

[0065] Figure 3 is a schematic diagram of the structure of a control device for an auxiliary robot provided in an embodiment of this application;

[0066] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0067] Figure 5 is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0069] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0070] As shown in Figure 1, which is a schematic diagram of a computer system provided in an embodiment of this application, the computer system includes a terminal device 101 and a server 102. The terminal device 101 has a client installed and running, and the server 102 provides background services to the client. The target object 103 can control the auxiliary robot using the terminal device 101, the client, or through interaction between the client and the server 102. That is, the control method for the auxiliary robot provided in this embodiment can be executed by the terminal device 101, by the server 102, or by both the terminal device 101 and the server 102; this embodiment does not limit the specific execution method.

[0071] In one possible implementation, server 102 undertakes the primary computational work, while terminal device 101 undertakes the secondary computational work. Alternatively, server 102 undertakes the secondary computational work, while terminal device 101 undertakes the primary computational work. Or, terminal device 101 and server 102 collaborate on computation using a distributed computing architecture.

[0072] Optionally, the terminal device 101 can be any electronic device product capable of human-computer interaction with the user through one or more methods such as a keyboard, touchpad, remote control, voice interaction, or handwriting device. For example, the terminal device 101 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, PC (Personal Computer), mobile phone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), smart car system, smart TV, etc.

[0073] Terminal device 101 can refer to one of a plurality of terminal devices. This embodiment uses terminal device 101 as an example. Those skilled in the art will know that the number of terminal devices 101 can be more or less. For example, there may be only one terminal device 101, or there may be dozens or hundreds of terminal devices 101, or more. This application embodiment does not limit the number or type of terminal devices 101.

[0074] Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: a cloud computing platform or a virtualization center. This embodiment of the application does not limit this. Server 102 communicates directly or indirectly with terminal device 101 via a wired or wireless network. Server 102 has data receiving, data processing, and data sending functions. Of course, server 102 may also have other functions, which are not limited in this embodiment of the application.

[0075] Those skilled in the art should understand that the terminal device 101 and server 102 described above are merely illustrative examples. Other existing or future terminal devices or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0076] This application provides a control method for an auxiliary robot, which can be applied to the aforementioned computer system. For ease of description, the terminal device 101 or server 102 executing the method of this application embodiment is collectively referred to as an electronic device; that is, the method of this application embodiment can be executed by an electronic device. Taking the flowchart of a control method for an auxiliary robot provided by this application embodiment shown in FIG2 as an example, as shown in FIG2, the method includes the following steps.

[0077] Step 201: Acquire a first image captured by the acquisition device. The first image represents the distance between the contact parts of the target object, the contact surface of the environment in which the target object is located, and the acquisition device.

[0078] The target object is a living organism or a mobile robot capable of movement; for example, the target object could be a human. The target object includes multiple supporting components and multiple movable components. Supporting components support the target object, enabling it to maintain its posture. Generally, supporting components may also have other functions. For example, some supporting components can cooperate to enable the target object to walk. Or, some supporting components can cooperate to enable the target object to grasp objects. Optionally, if the target object is a living organism, the supporting components are limb parts such as the thigh, lower leg, upper arm, lower arm, or waist; if the target object is a mobile robot, the supporting components can be a frame made of at least one material, such as a steel frame. Generally, the frame of a mobile robot is also called a skeleton. Supporting components are connected by movable components, which control the relative movement between the supporting components. For example, one end of the thigh and one end of the lower leg are connected by a knee joint, which controls the other end of the thigh and the other end of the lower leg to move closer or further apart. Optionally, if the target object is a living organism, the moving parts are joints such as the knee joint or hip joint; if the target object is a mobile robot, the moving parts can be movable connecting structures such as gears or couplings. Generally, the moving parts of a mobile robot can also be called joints. By controlling the movement of each supporting component through each moving part, it is possible to control the target object to perform movements such as walking or climbing.

[0079] The first image can be acquired in real time by an acquisition device, both when the target object is stationary and during its movement. The acquisition device is a distance-based image acquisition device; its type and acquisition principle are not limited here. Optionally, the acquisition device can be a depth sensor, an electronic device used to measure the distance between an object and the sensor, generating digital signals or images. Alternatively, the acquisition device can be a Time-of-Flight (ToF) camera. A ToF camera emits light signals and calculates the value of each pixel in the image by measuring the time it takes for the light signal to travel from emission to reflection back from the object. This allows for the acquisition of scene information based on the round-trip time of light, and is unaffected by lighting conditions. Based on this, embodiments of this application can use a ToF camera to acquire the first image in real time.

[0080] The first image includes the values ​​of multiple pixels, each representing the distance between the corresponding object region and the acquisition device. The object region can belong to a contact part of the target object, a contact surface of the environment in which the target object is located, or other objects in the environment such as buildings or plants. A contact surface is a surface in the environment that contacts the target object; for example, it could be the ground, a board, or a wall. A contact part is the portion of the target object that contacts the contact surface, and can include a supporting component or a portion of that component; for example, a contact part could include the target object's feet or hands. Generally, if the target object is walking, the contact part includes the feet, and the contact surface includes the ground; if the target object is climbing, the contact part includes the hands, and the contact surface includes a wall or slope.

[0081] Taking a ToF camera as an example, the ToF camera can capture real-time changes in the distance between the contacting part and the contact surface. For instance, by placing the ToF camera in a suitable position so that the acquisition range covers the contacting part and the contact surface during movement, changes in the contacting part and the contact surface can be clearly captured. The sampling rate and exposure parameters of the ToF camera can be set to ensure the real-time nature and accuracy of the acquisition.

[0082] At a certain moment, the acquisition device can acquire a first image, which is an image obtained by imaging the contact component and the contact surface, including the image area where the contact component is located and the image area where the contact surface is located. In some cases, the first image also includes the image area where other objects besides the contact component and the contact surface are located, such as the image area where trees, buildings, etc. are located.

[0083] Since the acquisition device is used to acquire the distance between the contact component, the contact surface, and the acquisition device, it can be deployed on the target object or in the environment, thus expanding the application scenarios of the method in this application embodiment. In related technologies, images are acquired using an RGB (three primary colors) camera, and the image quality is affected by lighting, thereby affecting the control of the target object. However, the first image in this application embodiment is not affected by lighting, which can improve the control accuracy of the target object.

[0084] Step 202: Segment a first region and a second region from the first image. The first region represents the distance between the contact component and the acquisition device, and the second region represents the distance between the contact surface and the acquisition device.

[0085] Image segmentation is a technique that divides an image into several regions and extracts regions of interest. Using image segmentation, a first region and a second region can be segmented from a first image. The first region is the image region where the contact component is located, including first information from multiple pixels. This first information reflects the distance between the region corresponding to the pixel on the contact component and the acquisition device. The second region is the image region where the contact surface is located, including second information from multiple pixels. This second information reflects the distance between the region corresponding to the pixel on the contact surface and the acquisition device.

[0086] In one possible implementation, step 202 includes steps 2021 to 2022 (not shown in the figure).

[0087] Step 2021: Preprocess the first image to obtain a preprocessed image. The preprocessing includes at least one of noise reduction, image cropping, and information filtering.

[0088] Denoising is a technique for removing noise from an image. It involves using filters, such as Gaussian or low-pass filters, to smooth the first image. By denoising the first image, noise can be removed, improving the segmentation results and reducing errors in subsequent image segmentation.

[0089] Image cropping is a technique for cropping images. In this example, the first image can be cropped according to a reference window to obtain the image region corresponding to the reference window. This image region includes the content of the contact parts and contact surfaces. The reference window can be a pre-set window or determined based on the first image. By cropping the first image, areas in the first image that do not belong to the contact parts and contact surfaces are removed, reducing the amount of data and improving the speed of subsequent image segmentation.

[0090] Information filtering is a filtering technique. In the example, the first image includes the values ​​of multiple pixels (referred to as pixel values), where each pixel value represents the distance between the region corresponding to the pixel and the acquisition device. Pixel values ​​in the first image that are not within a reference range can be filtered out, while those within the reference range are retained. The retained pixel values ​​include the values ​​of each pixel in the image region where the contact surface is located and the values ​​of each pixel in the image region where the contact component is located. The reference range can be pre-set or randomly generated data, or it can be a value input by the target object. By performing information filtering on the first image, pixel values ​​unrelated to the contact component and contact surface can be filtered out, while pixel values ​​related to the contact component and contact surface can be retained, reducing the amount of data and improving the speed of subsequent image segmentation.

[0091] In practical applications, preprocessing can also include other processes, such as adjusting the size of the first image or compressing it. When there are multiple preprocessing methods, the order of each preprocessing step is not limited. For example, the first image can be cropped, filtered, and denoised sequentially.

[0092] Step 2022: Segment the first region and the second region from the preprocessed image.

[0093] Image segmentation technology is used to segment a first region and a second region from a preprocessed image. Because the preprocessed image has characteristics such as small data volume and low noise, image segmentation is fast, and the accuracy of the first and second regions is high. This application does not limit the image segmentation method.

[0094] In an exemplary embodiment, step 2022 includes: performing image segmentation on the preprocessed image using an image segmentation model to obtain a first mask image and a second mask image, wherein the first mask image is used to describe whether each pixel in the preprocessed image belongs to a contact part, and the second mask image is used to describe whether each pixel belongs to a contact surface; determining a first region based on the first mask image; and determining a second region based on the second mask image.

[0095] Image segmentation models are learning models, and their structure and size are not limited here. For example, an image segmentation model can be a U-Net model. A U-Net model includes an encoder (also called a shrinking path) and a decoder (also called an expanding path), connected by skip connections, forming a U-shape. The encoder is responsible for downsampling and feature extraction, while the decoder is used for image size restoration and segmentation. Skip connections are used to concatenate the feature maps of the encoder and decoder to retain more detailed information and improve segmentation accuracy.

[0096] Generally, a pre-trained model can be obtained, which is a model with image segmentation capabilities. A large number of sample images can be acquired, representing the distances between the contact parts, contact surfaces, and the device that acquired the sample images. The pre-trained model is trained using these sample images to obtain the image segmentation model, which can accurately segment the image regions where the contact parts and contact surfaces are located, with high segmentation accuracy and high training efficiency.

[0097] Optionally, data augmentation, such as rotation, scaling, and translation, can be applied to the sample images. The augmented images are then used to train a pre-trained model to obtain an image segmentation model, thereby improving the model's segmentation accuracy for image regions containing contact parts and contact surfaces at different angles and distances. The implementation principles of training a pre-trained model using sample images and training a pre-trained model using data-augmented images are similar. Only one implementation principle is described below; the other implementation principle can be found in this explanation.

[0098] In this example, each sample image can be labeled to obtain the labeling results. The labeling results include the image region where the contact part is located and the image region where the contact surface is located. For example, the image region where the contact part is located is labeled as 1, the image region where the contact surface is located is labeled as 0, and other regions are labeled as 2. Next, the sample images are input into the pre-trained model, which performs image segmentation on the sample images to obtain prediction results. The prediction results include the image regions where the contact part is located and the image regions where the contact surface is located. Then, based on the labeling and prediction results of each sample image, a loss value is determined according to the loss function, and the pre-trained model is trained based on the loss value. The type of loss function is not limited here; for example, the loss function includes at least one of the following: cross-entropy loss function, Dice loss function, etc. The pre-trained model can be trained multiple times in the above manner to obtain the image segmentation model. The number of training times can be a pre-set number or a number of times to ensure that the model accuracy is not less than a threshold. Generally, a test set can be used to evaluate the model accuracy. The test set includes multiple test images, which represent the distances between the contact part, the contact surface, and the device that acquired the test images, respectively. The evaluation process will not be elaborated here.

[0099] Optionally, the image segmentation model is a single-task model. In this case, a segmentation model for segmenting the image region where the contact component is located can be trained, and a first mask image can be obtained based on the first image using this model. Alternatively, a segmentation model for segmenting the image region where the contact surface is located can be trained, and a second mask image can be obtained based on the first image using this model. Or, the image segmentation model is a multi-task model. In this case, the first image is input into the image segmentation model to obtain a first mask image and a second mask image.

[0100] The first mask image is used to describe whether each pixel in the preprocessed image belongs to a contact part. For example, in the first mask image, if the mask value of a pixel is 1, it indicates that the pixel belongs to a contact part; if the mask value of a pixel is 0, it indicates that the pixel does not belong to a contact part. Pixels belonging to contact parts in the first mask image can be extracted to obtain the first region. Alternatively, the first mask image can be post-processed, such as dilation or erosion, and the first region can be determined based on the post-processed image. Dilation is used to increase the area of ​​the image region belonging to the contact part, while erosion is used to remove noise, segment connected regions, and reduce the size of the image region belonging to the contact part. Through post-processing, edges can be smoothed, small regions filtered out, and noise reduced, improving the accuracy of the first region.

[0101] Similarly, the second mask image is used to describe whether each pixel belongs to the contact surface. For example, in the second mask image, if the mask value of a pixel is 1, it indicates that the pixel belongs to the contact surface; if the mask value of a pixel is 0, it indicates that the pixel does not belong to the contact surface. The second region can be determined based on the second mask image. The method for determining the second region is similar to that for the first region, and will not be repeated here.

[0102] Image segmentation is performed by an image segmentation model, enabling electronic devices to automatically identify the image regions where contact components and contact surfaces are located, facilitating subsequent judgment of the contact state.

[0103] Step 203: Determine the first contact state of the target object based on the first region and the second region. The first contact state characterizes whether the contact component is in contact with the contact surface.

[0104] In this embodiment, the first region includes first information, which includes the values ​​of each pixel belonging to the contact component, and the pixel values ​​are the distances between the corresponding areas on the contact component and the acquisition device. The second region includes second information, which includes the values ​​of each pixel belonging to the contact surface, and the pixel values ​​are the distances between the corresponding areas on the contact surface and the acquisition device.

[0105] The first contact state of the target object can be determined using the values ​​included in the first information and the values ​​included in the second information. The first contact state characterizes whether the contacting component is in contact with the contact surface. The method for determining the first contact state is not limited here.

[0106] In an exemplary embodiment, step 203 includes: determining the distance between the contact component and the contact surface based on the first region and the second region; if the distance is less than a threshold, determining a first contact state characterizing that the contact component is in contact with the contact surface; if the distance is not less than the threshold, determining that the first contact state characterizes that the contact component is not in contact with the contact surface.

[0107] In this embodiment, the average value of each value included in the first information can be calculated to obtain a first value; the average value of each value included in the second information can be calculated to obtain a second value. The difference between the first value and the second value is used as the distance between the contact component and the contact surface.

[0108] The distance threshold THRESHOLD can be obtained. This threshold can be a value set based on human experience, a value obtained through experimental verification, or it can be related to the environment of the target object. That is, a corresponding distance threshold can be set according to different application scenarios; for example, THRESHOLD = 0.05m. If the distance between the contacting component and the contact surface is less than the distance threshold, the first contact state is determined, representing contact with the contact surface; if the distance between the contacting component and the contact surface is not less than the distance threshold, the first contact state is determined, representing non-contact with the contact surface.

[0109] The method according to the embodiments of this application can quickly determine whether the contacting component is in contact with the contacting surface without complex calculations, reducing the amount of movement and improving the real-time performance of control.

[0110] Step 204: Control the auxiliary robot according to the first contact state. The auxiliary robot is used to assist the target object in moving.

[0111] During the movement of a target object, its posture and behavioral characteristics change periodically. These posture and behavioral characteristics within one cycle can be referred to as reference motion information. For example, during the movement of the target object, contact surfaces and non-contact surfaces alternate, and the reference motion cycle can reflect this alternation. Based on the initial contact state and the reference motion information, an auxiliary robot can be controlled to assist the target object's movement.

[0112] An assistive robot is a type of robot that assists a target object in its movement. Generally, an assistive robot can be attached to the exterior of a target object to enhance its strength, agility, and endurance. For example, an assistive robot can be attached to the legs of a target object to enhance leg strength and assist in walking. Or, an assistive robot can be attached to the arms of a target object to enhance arm strength and assist in climbing.

[0113] Optionally, if the assistive robot assists the target object in walking, the reference motion information is the posture and behavioral characteristics of the target object walking within one cycle; that is, the reference motion information is gait information. If the assistive robot assists the target object in climbing, the reference motion information is the posture and behavioral characteristics of the target object climbing within one cycle; that is, the reference motion information is climbing information. There are several ways to obtain reference motion information. For example, reference motion information can be set based on human experience, or it can be obtained according to implementation methods A or B below. Implementation methods A and B are described below.

[0114] In implementation method A, before step 204, the method further includes: acquiring multiple consecutive frames of second images captured by a data acquisition device, wherein the second images represent the distances between the contact component, the contact surface, and the data acquisition device; determining multiple second contact states based on the multiple frames of second images; and determining first motion information of the target object based on the multiple second contact states, wherein the first motion information is used to characterize the posture and behavioral features of the target object during movement. In this case, step 204 includes: controlling the auxiliary robot based on the first motion information and the first contact states.

[0115] In this example, the reference motion information includes first motion information, the determination of which depends on multiple consecutive frames of second images. Multiple frames of second images can be acquired in real-time during the movement of the target object using an acquisition device. Similar to the first image, the second image is an image obtained by imaging the contact component and the contact surface, including the image area where the contact component is located and the image area where the contact surface is located. Optionally, the second image may also include image areas of other objects besides the contact component and the contact surface.

[0116] Following the implementation principles of steps 202 to 203, a second contact state characterizing whether a contacting component is in contact with a contact surface can be determined based on any frame of the second image. The determination process will not be elaborated here. First motion information is determined by analyzing multiple second contact states. For example, for a second contact state characterizing contact between a contacting component and a contact surface, the start time (or start frame number) and end time (or end frame number) of contact are recorded, and a first duration of contact is determined based on the recorded data. Similarly, for a second contact state characterizing when a target object is not in contact with a contact surface, the start time (or start frame number) and end time (or end frame number) of non-contact are recorded, and a second duration of non-contact is determined based on the recorded data. First motion information is determined based on the first and second durations.

[0117] Based on this, the electronic device can generate control parameters for the auxiliary robot according to the first motion information and the first contact state, and control the auxiliary robot according to the control parameters to achieve the movement of the target object assisted by the auxiliary robot. The control parameters include, but are not limited to, at least one of speed, acceleration, steering angle, angular velocity, etc. It can be understood that the electronic device can execute steps 201 to 204 in real time, thereby achieving the movement of the target object assisted by the auxiliary robot by alternately appearing in contact with the contact surface and not in contact with the contact surface.

[0118] In practical applications, electronic devices can display the target object's initial contact state, motion information, and other data in real time. For example, they can display the duration of contact and motion. Furthermore, electronic devices can store data generated during the target object's control process in real time, such as the initial contact state and image segmentation results. Storing this data facilitates subsequent data analysis and improves the robot's assistive effect on the target object.

[0119] Furthermore, the electronic device can determine whether the contacting component is in an abnormal motion state based on the first motion information and the first contact state. If the target object is in an abnormal motion state, such as when at least one of the following occurs: the time of contact with the contact surface, the time of non-contact with the contact surface, or the motion cycle, the electronic device can generate a warning message to alert the target object to the abnormal motion.

[0120] In implementation method B, the target object includes multiple supporting components and multiple movable components, with the movable components used to control the supporting components. Before step 204, the method further includes: acquiring first motion data of the multiple supporting components and second motion data of the multiple movable components; determining second motion information of the target object based on the first and second motion data, the second motion information being used to characterize the posture and behavioral features of the target object during movement. In this case, step 204 includes: controlling the auxiliary robot based on the second motion information and the first contact state.

[0121] In this embodiment, the reference motion information includes second motion information, the determination of which depends on the motion data of the support component and the motion data of the movable component. A first motion sensor can be attached to the support component to collect first motion data, and a second motion sensor can be attached to the movable component to collect second motion data. Alternatively, when the assistive robot is a wearable exoskeleton device, a first motion sensor can be attached to the support component of the exoskeleton device to collect first motion data, and a second motion sensor can be attached to the movable component of the exoskeleton device to collect second motion data.

[0122] Motion sensors primarily trigger relevant response mechanisms by detecting the motion of objects. These sensors typically employ infrared, ultrasonic, and microwave radar technologies to achieve motion detection. This application does not limit the type of motion sensor; for example, motion sensors include velocity sensors, pressure sensors, angular velocity sensors, orientation angle sensors, and angle sensors. Therefore, this application does not limit the type or quantity of the first and second motion sensors. Since different motion sensors can measure different motion data—for example, a velocity sensor can measure velocity data, while a pressure sensor can measure pressure data—this application also does not limit the type or format of the first and second motion data.

[0123] For example, an Inertial Measurement Unit (IMU) sensor is installed on the waist, left thigh, right thigh, left calf, and right calf of the target object. For example, an IMU sensor is installed on the waist assembly, thigh assembly, and calf assembly of the exoskeleton device. An IMU sensor is an electronic device integrating an accelerometer and a gyroscope, used to measure data such as the object's acceleration, angular velocity, and orientation angle. Since the first motion sensor includes an IMU sensor, the first motion data includes acceleration data, angular velocity data, and orientation angle data. Optionally, the acceleration data and angular velocity data include sub-data for the three directional axes, used to describe the motion state of the corresponding support component along the three directional axes.

[0124] In addition, an angle sensor / IMU can be attached to both the hip and knee joints of the target object to measure the angle data of the corresponding joints. Alternatively, an angle sensor / IMU can be installed on both the hip and knee joint components of the exoskeleton device. That is, the second motion sensor includes the angle sensor / IMU, and the second motion data includes angle data. The hip joint, located between the waist and thigh, is used to control the relative movement between the waist and thigh; therefore, the angle data of the hip joint can reflect the relative movement between the waist and thigh. Similarly, the knee joint, located between the thigh and lower leg, is used to control the relative movement between the thigh and lower leg; therefore, the angle data of the knee joint can reflect the relative movement between the thigh and lower leg.

[0125] It should be noted that the first and second motion sensors can acquire motion data in real time. The acquisition frequency can be flexibly set according to the application scenario. For example, the sampling frequency of both the IMU sensor and the angle sensor is 500Hz to ensure real-time acquisition of motion data. The data acquired by the motion sensors can be transmitted to an electronic device wirelessly or via wired connection, allowing the electronic device to analyze each piece of first and second motion data to determine the second motion information. The determination method is not limited here. One possible implementation is shown below.

[0126] In an exemplary embodiment, second motion information of the target object is determined based on a plurality of first motion data and a plurality of second motion data, including steps B1 to B3 (not shown in the figure).

[0127] Step B1: Determine the attitude information of multiple support components based on the first motion data.

[0128] In this embodiment, the attitude information of multiple support components can be calculated based on an attitude calculation algorithm and multiple first motion data. The attitude information of the support components reflects their attitude, such as rotation direction and angle with a reference surface. Taking the first motion data, which includes acceleration and angular velocity data, as an example, a three-dimensional attitude angle can be calculated based on a quaternion attitude calculation algorithm, acceleration data, and angular velocity data. This three-dimensional attitude angle reflects the rotation direction and angle of the support component on three directional axes. In this way, the three-dimensional attitude angle corresponding to each support component can be calculated.

[0129] Step B2: Based on multiple posture information and second motion data, determine multiple relative motion information. One relative motion information is used to describe the motion of one support component relative to another support component.

[0130] In this embodiment, data fusion algorithms such as Kalman filtering can be used to fuse multiple posture information with second motion data to obtain fused data. Multiple relative motion information is then determined based on this fused data. By fusing posture information and multiple second motion data, the various motion data are made consistent in time and space, improving the accuracy of motion capture. By determining multiple posture information, the relative motion between support components can be determined. For example, based on the fused data, the motion of the thigh relative to the waist, and the motion of the lower leg relative to the thigh, can be determined. Multiple relative motion conditions can reflect complete motion information, including angular changes between support components and the spatial trajectory formed by the motion.

[0131] Step B3: Determine the second motion information of the target object based on multiple relative motion information.

[0132] In this embodiment, the motion trajectory of a target object can be constructed based on multiple relative motion information, indicating the trajectory formed by the target object at different stages of the motion process. For example, the motion trajectory can indicate the trajectory of the target object when lifting its leg, the trajectory of the leg swinging in space, and the trajectory of the leg touching the ground. Based on the motion trajectory, second motion information is determined. Taking gait information as an example, the gait information includes at least one of the following parameters: stride length, stride frequency, rate of change of the angle of the moving part, time of contact with the contact surface (referred to as ground contact time), time of non-contact with the contact surface (referred to as swing time), etc.

[0133] In an exemplary embodiment, second motion information of the target object is determined based on a plurality of first motion data and a plurality of second motion data, including steps B4 to B5 (not shown in the figure).

[0134] Step B4: Preprocess the first motion data and the second motion data to obtain preprocessed data. The preprocessing includes at least one of noise reduction processing and calibration processing.

[0135] Since motion data is susceptible to environmental noise and minor jitter, denoising processing can be performed on both the first and second motion data to smooth the data and reduce noise interference. The denoising method is not limited here; for example, a low-pass filter or a Kalman filter can be used to denoise the first and second motion data.

[0136] Generally, multiple motion sensors are needed to collect first and second motion data. Therefore, the first and second motion data need to be calibrated to ensure they are relative to the same reference coordinate system. This means that calibrating each motion sensor to its zero point eliminates deviations in the initial state (i.e., the output state without external input), improving the accuracy of the first and second motion data. Furthermore, since moving parts control supporting parts, and these supporting parts can also affect moving parts—for example, the hip joint controls the relative movement between the waist and thigh, which affects the hip joint angle—the accuracy and stability of the motion data can be improved by calibrating the first motion data with the second motion data and vice versa. The calibration method is not limited here.

[0137] In practical applications, preprocessing can also include other processing methods, which will not be elaborated here. When there are multiple preprocessing methods, the order of the preprocessing is not limited. For example, the first motion data and the second motion data can be denoised and calibrated in sequence.

[0138] Step B5: Based on the preprocessed data, determine the second motion information of the target object.

[0139] It is understandable that step B5 can be implemented according to the implementation principles shown in steps B1 to B3, and will not be elaborated further here.

[0140] In practical applications, motion data can be collected in real time using motion sensors, and second motion information can be determined in real time. Based on at least one of the first and second motion information, and the first contact state, control parameters for the auxiliary robot are generated. For example, averaging or weighted calculations can be performed on the first and second motion information to obtain target motion information. Control parameters are then generated based on the target motion information and the first contact state. The weights of the first and second motion information can be set based on human experience. Subsequently, the auxiliary robot can be controlled according to the control parameters, thereby enabling the robot to assist the target object in its movement.

[0141] Understandably, electronic devices can determine whether a target object is in an abnormal motion state. For example, the motion state can be determined based on the angular change rate (such as the angular change rate of the hip joint, the angular change rate of the knee joint, etc.) in the second motion information. Different control parameters are generated for different motion states. For instance, when the target object is in an abnormal motion state, the control parameters can instruct the assist robot to provide auxiliary torque to help the target object maintain stable motion.

[0142] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the images involved in this application were all obtained with full authorization.

[0143] In the above method, a first image is acquired using a data acquisition device, and a first region and a second region are segmented from the first image. Since the first region represents the distance between the contacting component of the target object and the data acquisition device, and the second region represents the distance between the contact surface of the target object's environment and the data acquisition device, it is possible to determine whether the contacting component is in contact with the contact surface based on the first and second regions, thus obtaining the first contact state. During the target object's movement, it will periodically alternate between contacting the contacting surface and not contacting it. Based on this, the auxiliary robot can be controlled according to the first contact state to assist the target object in completing movements such as walking and climbing.

[0144] Figure 3 is a schematic diagram of the structure of a control device for an auxiliary robot provided in an embodiment of this application. As shown in Figure 3, the device includes:

[0145] The acquisition module 301 is used to acquire a first image acquired by the acquisition device. The first image represents the distance between the contact parts of the target object, the contact surface of the environment in which the target object is located, and the acquisition device.

[0146] The segmentation module 302 is used to segment a first region and a second region from the first image. The first region represents the distance between the contact component and the acquisition device, and the second region represents the distance between the contact surface and the acquisition device.

[0147] The determining module 303 is used to determine the first contact state of the target object based on the first region and the second region. The first contact state characterizes whether the contact component is in contact with the contact surface.

[0148] The control module 304 is used to control the auxiliary robot according to the first contact state. The auxiliary robot is used to assist the target object in moving.

[0149] In one possible implementation, the segmentation module 302 is used to preprocess the first image to obtain a preprocessed image, the preprocessing including at least one of noise reduction, image cropping and information filtering; and to segment a first region and a second region from the preprocessed image.

[0150] In one possible implementation, the segmentation module 302 is used to segment the preprocessed image using an image segmentation model to obtain a first mask image and a second mask image. The first mask image is used to describe whether each pixel in the preprocessed image belongs to a contact part, and the second mask image is used to describe whether each pixel belongs to a contact surface. Based on the first mask image, a first region is determined; based on the second mask image, a second region is determined.

[0151] In one possible implementation, the determining module 303 is used to determine the distance between the contact component and the contact surface based on the first region and the second region; if the distance is less than a threshold, a first contact state is determined to indicate that the contact component is in contact with the contact surface; if the distance is not less than the threshold, the first contact state is determined to indicate that the contact component is not in contact with the contact surface.

[0152] In one possible implementation, the acquisition module 301 is further configured to acquire multiple consecutive frames of second images acquired by the acquisition device, wherein the second images represent the distances between the contact component, the contact surface and the acquisition device, respectively.

[0153] The determining module 303 is further configured to determine multiple second contact states based on multiple frames of second images; and to determine first motion information of the target object based on the multiple second contact states, wherein the first motion information is used to characterize the posture and behavior features of the target object during motion.

[0154] The control module 304 is used to control the auxiliary robot based on the first motion information and the first contact state.

[0155] In one possible implementation, the target object includes multiple supporting components and multiple movable components, the movable components being used to control the supporting components;

[0156] The acquisition module 301 is also used to acquire first motion data of multiple support components and second motion data of multiple movable components;

[0157] The determining module 303 is also used to determine the second motion information of the target object based on the first motion data and the second motion data. The second motion information is used to characterize the posture and behavior features of the target object during motion.

[0158] The control module 304 is used to control the auxiliary robot based on the second motion information and the first contact state.

[0159] In one possible implementation, the determining module 303 is used to preprocess the first motion data and the second motion data to obtain preprocessed data, wherein the preprocessing includes at least one of noise reduction processing and calibration processing; and based on the preprocessed data, the second motion information of the target object is determined.

[0160] In one possible implementation, the determining module 303 is used to determine the attitude information of multiple support components based on the first motion data; determine multiple relative motion information based on the multiple attitude information and the second motion data, wherein one relative motion information is used to describe the motion of one support component relative to another support component; and determine the second motion information of the target object based on the multiple relative motion information.

[0161] In the aforementioned device, a first image is acquired by a data acquisition device, and a first region and a second region are segmented from the first image. Since the first region represents the distance between the contacting component of the target object and the data acquisition device, and the second region represents the distance between the contact surface of the target object's environment and the data acquisition device, it is possible to determine whether the contacting component is in contact with the contact surface based on the first and second regions, thus obtaining the first contact state. During the target object's movement, it will periodically alternate between contacting the contacting surface and not contacting it. Based on this, an auxiliary robot can be controlled according to the first contact state to assist the target object in completing movements such as walking and climbing.

[0162] It should be understood that the device shown in Figure 3 above is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0163] Figure 4 shows a structural block diagram of an electronic device 400 provided in an exemplary embodiment of this application. Optionally, the electronic device 400 is a terminal device. The electronic device 400 includes a processor 401 and a memory 402.

[0164] Processor 401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0165] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 are used to store at least one computer program, which is executed by the processor 401 to implement the control method for the auxiliary robot provided in the method embodiments of this application.

[0166] In some embodiments, the electronic device 400 may also optionally include a peripheral device interface 403 and at least one peripheral device. The processor 401, memory 402, and peripheral device interface 403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 404, a display screen 405, a camera assembly 406, an audio circuit 407, and a power supply 408.

[0167] Peripheral device interface 403 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 401 and memory 402. In some embodiments, processor 401, memory 402 and peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 401, memory 402 and peripheral device interface 403 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0168] The radio frequency (RF) circuit 404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 404 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 404 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 404 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 404 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0169] Display screen 405 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 405 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 401 for processing. In this case, display screen 405 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 405, disposed on the front panel of electronic device 400; in other embodiments, there may be at least two display screens, disposed on different surfaces of electronic device 400 or in a folded design; in other embodiments, display screen 405 may be a flexible display screen, disposed on a curved or folded surface of electronic device 400. Furthermore, display screen 405 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 405 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0170] The camera assembly 406 is used to acquire images or videos. Optionally, the camera assembly 406 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 406 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0171] The audio circuit 407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 401 for processing, or input to the radio frequency circuit 404 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the electronic device 400. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 401 or the radio frequency circuit 404 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 407 may also include a headphone jack.

[0172] Power supply 408 is used to supply power to various components in electronic device 400. Power supply 408 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 408 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0173] In some embodiments, the electronic device 400 further includes one or more sensors 409. The one or more sensors 409 include, but are not limited to, an accelerometer 411, a gyroscope 412, a pressure sensor 413, an optical sensor 414, and a proximity sensor 415.

[0174] Accelerometer 411 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by electronic device 400. For example, accelerometer 411 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 401 can control display screen 405 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 411. Accelerometer 411 can also be used for games or for acquiring user motion data.

[0175] The gyroscope sensor 412 can detect the orientation and rotation angle of the electronic device 400. The gyroscope sensor 412 can work in conjunction with the accelerometer sensor 411 to collect 3D motion data from the user on the electronic device 400. Based on the data collected by the gyroscope sensor 412, the processor 401 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0176] The pressure sensor 413 can be disposed on the side bezel of the electronic device 400 and / or the lower layer of the display screen 405. When the pressure sensor 413 is disposed on the side bezel of the electronic device 400, it can detect the user's grip signal on the electronic device 400, and the processor 401 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 413. When the pressure sensor 413 is disposed on the lower layer of the display screen 405, the processor 401 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 405. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0177] Optical sensor 414 is used to collect ambient light intensity. In one embodiment, processor 401 can control the display brightness of display screen 405 based on the ambient light intensity collected by optical sensor 414. Specifically, when the ambient light intensity is high, the display brightness of display screen 405 is increased; when the ambient light intensity is low, the display brightness of display screen 405 is decreased. In another embodiment, processor 401 can also dynamically adjust the shooting parameters of camera assembly 406 based on the ambient light intensity collected by optical sensor 414.

[0178] A proximity sensor 415, also known as a distance sensor, is typically mounted on the front panel of an electronic device 400. The proximity sensor 415 is used to detect the distance between the user and the front of the electronic device 400. In one embodiment, when the proximity sensor 415 detects that the distance between the user and the front of the electronic device 400 is gradually decreasing, the processor 401 controls the display screen 405 to switch from a screen-on state to a screen-off state; when the proximity sensor 415 detects that the distance between the user and the front of the electronic device 400 is gradually increasing, the processor 401 controls the display screen 405 to switch from a screen-off state to a screen-on state.

[0179] Those skilled in the art will understand that the structure shown in FIG4 does not constitute a limitation on the electronic device 400, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0180] Figure 5 is a schematic diagram of the server structure provided in the embodiment of this application. The server 500 can vary considerably due to different configurations or performance. It may include one or more processors 501 and one or more memories 502. The one or more memories 502 store at least one computer program, which is loaded and executed by the one or more processors 501 to implement the control method of the auxiliary robot provided in the above-described method embodiments. For example, the processor 501 is a CPU. Of course, the server 500 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 500 may also include other components for implementing device functions, which will not be elaborated here.

[0181] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described control methods for an auxiliary robot.

[0182] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0183] In an exemplary embodiment, a computer program is also provided, which is at least one such computer program, loaded and executed by a processor, to enable an electronic device to implement any of the above-described control methods for an auxiliary robot.

[0184] In an exemplary embodiment, a computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described auxiliary robot control methods.

[0185] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0186] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0187] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A control method for an auxiliary robot, wherein, The method includes: A first image is acquired by the acquisition device, wherein the first image represents the distance between the contact parts of the target object, the contact surface of the environment in which the target object is located, and the acquisition device. A first region and a second region are segmented from the first image. The first region represents the distance between the contact component and the acquisition device, and the second region represents the distance between the contact surface and the acquisition device. Based on the first region and the second region, a first contact state of the target object is determined, wherein the first contact state characterizes whether the contact component contacts the contact surface; The auxiliary robot is controlled according to the first contact state, and the auxiliary robot is used to assist the target object in moving.

2. The method according to claim 1, wherein, The step of segmenting the first region and the second region from the first image includes: The first image is preprocessed to obtain a preprocessed image, wherein the preprocessing includes at least one of noise reduction, image cropping, and information filtering; The first region and the second region are segmented from the preprocessed image.

3. The method according to claim 2, wherein, The step of segmenting the first region and the second region from the preprocessed image includes: The preprocessed image is segmented using an image segmentation model to obtain a first mask image and a second mask image. The first mask image is used to describe whether each pixel in the preprocessed image belongs to the contact component, and the second mask image is used to describe whether each pixel belongs to the contact surface. Based on the first mask image, determine the first region; The second region is determined based on the second mask image.

4. The method according to any one of claims 1 to 3, wherein, Determining the first contact state of the target object based on the first region and the second region includes: The distance between the contact component and the contact surface is determined based on the first region and the second region; If the distance is less than a threshold, the first contact state is determined to represent that the contact component is in contact with the contact surface; If the distance is not less than the threshold, the first contact state is determined to indicate that the contact component is not in contact with the contact surface.

5. The method according to any one of claims 1 to 4, wherein, The method further includes: Acquire a series of second images captured by the acquisition device, wherein the second images represent the distances between the contact component, the contact surface, and the acquisition device, respectively; Multiple second contact states are determined based on the multiple frames of the second images; First motion information of the target object is determined based on the plurality of second contact states, and the first motion information is used to characterize the posture and behavioral features of the target object during motion. The step of controlling the auxiliary robot according to the first contact state includes: The robot is controlled based on the first motion information and the first contact state.

6. The method according to any one of claims 1 to 4, wherein, The target object includes multiple supporting components and multiple movable components, wherein the movable components are used to control the supporting components, and the method further includes: Acquire first motion data of the plurality of supporting components and second motion data of the plurality of movable components; Based on the first motion data and the second motion data, second motion information of the target object is determined, wherein the second motion information is used to characterize the posture and behavioral features of the target object during motion; The step of controlling the auxiliary robot according to the first contact state includes: The robot is controlled based on the second motion information and the first contact state.

7. The method according to claim 6, wherein, The step of determining the second motion information of the target object based on the first motion data and the second motion data includes: The first motion data and the second motion data are preprocessed to obtain preprocessed data, wherein the preprocessing includes at least one of noise reduction processing and calibration processing. Based on the preprocessed data, the second motion information of the target object is determined.

8. The method according to claim 6, wherein, The step of determining the second motion information of the target object based on the first motion data and the second motion data includes: Based on the first motion data, determine the attitude information of the plurality of support components; Based on the multiple posture information and the second motion data, multiple relative motion information is determined, and one relative motion information is used to describe the motion of one support component relative to another support component; Based on the plurality of relative motion information, the second motion information of the target object is determined.

9. A control device for an auxiliary robot, wherein, The device includes: The acquisition module is used to acquire a first image captured by the acquisition device, wherein the first image represents the distance between the contact parts of the target object, the contact surface of the environment in which the target object is located, and the acquisition device. A segmentation module is used to segment a first region and a second region from the first image, wherein the first region represents the distance between the contact component and the acquisition device, and the second region represents the distance between the contact surface and the acquisition device; The determining module is configured to determine a first contact state of the target object based on the first region and the second region, wherein the first contact state characterizes whether the contact component contacts the contact surface; A control module is used to control an auxiliary robot based on the first contact state, the auxiliary robot being used to assist the target object in its movement.

10. The apparatus according to claim 9, wherein, The segmentation module is used to preprocess the first image to obtain a preprocessed image. The preprocessing includes at least one of noise reduction, image cropping, and information filtering. The first region and the second region are segmented from the preprocessed image.

11. The apparatus according to claim 10, wherein, The segmentation module is used to segment the preprocessed image using an image segmentation model to obtain a first mask image and a second mask image. The first mask image is used to describe whether each pixel in the preprocessed image belongs to the contact component, and the second mask image is used to describe whether each pixel belongs to the contact surface. A first region is determined based on the first mask image, and a second region is determined based on the second mask image.

12. The apparatus according to any one of claims 9 to 10, wherein, The determining module is configured to determine the distance between the contact component and the contact surface based on the first region and the second region; if the distance is less than a threshold, determine the first contact state to indicate that the contact component is in contact with the contact surface; if the distance is not less than the threshold, determine the first contact state to indicate that the contact component is not in contact with the contact surface.

13. The apparatus according to any one of claims 9 to 12, wherein, The acquisition module is further configured to acquire multiple consecutive frames of second images acquired by the acquisition device, wherein the second images represent the distances between the contact component, the contact surface and the acquisition device, respectively; The determining module is further configured to determine multiple second contact states based on the multiple frames of the second image; and to determine first motion information of the target object based on the multiple second contact states, wherein the first motion information is used to characterize the posture and behavioral features of the target object during motion; The control module is used to control the auxiliary robot based on the first motion information and the first contact state.

14. The apparatus according to any one of claims 9 to 12, wherein, The target object includes multiple supporting components and multiple movable components, wherein the movable components are used to control the supporting components; The acquisition module is further configured to acquire first motion data of the plurality of support components and second motion data of the plurality of movable components; The determining module is further configured to determine second motion information of the target object based on the first motion data and the second motion data, wherein the second motion information is used to characterize the posture and behavioral features of the target object during motion; The control module is used to control the auxiliary robot based on the second motion information and the first contact state.

15. The apparatus according to claim 14, wherein, The determining module is used to preprocess the first motion data and the second motion data to obtain preprocessed data, wherein the preprocessing includes at least one of noise reduction processing and calibration processing. Based on the preprocessed data, the second motion information of the target object is determined.

16. The apparatus according to claim 14, wherein, The determining module is configured to determine the posture information of the plurality of support components based on the first motion data; determine the plurality of relative motion information based on the plurality of posture information and the second motion data, wherein one relative motion information describes the motion of one support component relative to another support component; and determine the second motion information of the target object based on the plurality of relative motion information.

17. An electronic device, wherein, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the control method for the auxiliary robot as described in any one of claims 1 to 8.

18. A computer-readable storage medium, wherein, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the control method for the auxiliary robot as described in any one of claims 1 to 8.

19. A computer program product, wherein, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the control method for the auxiliary robot as described in any one of claims 1 to 8.

Images