Control method and apparatus, device, computer-readable storage medium, and computer program product

Abstract

A control method and apparatus, a device, a computer-readable storage medium, and a computer program product, the method comprising: collecting target visual information for a target object, and acquiring a reference feature vector, the reference feature vector being used for characterizing features of a reference joint angle sequence, the reference joint angle sequence comprising angles of joints of an electronic device when grabbing a reference object, and an object type of the reference object being identical to an object type of the target object; on the basis of the reference feature vector and the target visual information, predicting joint information of the electronic device to obtain a predicted joint information sequence, the predicted joint information sequence comprising a plurality of predicted joint vectors, and the predicted joint vectors comprising prediction results for the joints in the electronic device; and on the basis of each predicted joint vector in the predicted joint information sequence, controlling the joints of the electronic device to move, so as to grab the target object. The present application can improve the accuracy and efficiency of object grabbing.

Description

Control methods, apparatus, equipment, computer-readable storage media and computer program products

[0001] This application claims priority to Chinese Patent Application No. 202411749228.0, filed on November 29, 2024, entitled "Control Method, Apparatus, Device, Computer-Readable Storage Medium and Computer Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of robot control, and more particularly to a control method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0003] Robotic grasping refers to the process by which a robot uses a gripper or robotic arm to grasp, move, manipulate, and release objects. The core task of grasping is to enable the robot to flexibly and accurately select target objects from the environment and perform operations, similar to the function of a human hand.

[0004] In industrial manufacturing and assembly, robots can precisely and quickly grasp, transport, and manipulate parts or finished products around the clock, replacing manual handling work, reducing production time and error rates, and thus significantly improving production efficiency. In the logistics and warehousing industry, robots can efficiently process large volumes of orders without interruption, enabling the sorting, transporting, loading, and unloading of goods, thereby improving the level of automation in goods handling.

[0005] In related technologies, there are issues with the robot's object grasping process, such as the inability to accurately grasp objects or unreasonable grasping processes, resulting in a low grasping success rate. Technical issues

[0006] This application provides a control method, apparatus, device, computer-readable storage medium, and computer program product that can improve the accuracy and efficiency of object grasping. Technical solutions

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides a control method, the method comprising:

[0009] Visual information of the target object is collected, and a reference feature vector is obtained. The reference feature vector is used to characterize the features of the reference joint angle sequence. The reference joint angle sequence includes the angles of each joint during the process of the electronic device grasping the reference object. The object type of the reference object is the same as the object type of the target object.

[0010] Based on the reference feature vector and the target visual information, the joint information of the electronic device is predicted to obtain a predicted joint information sequence. The predicted joint information sequence includes multiple predicted joint vectors, and the predicted joint vectors include the prediction results of each joint in the electronic device.

[0011] Based on each predicted joint vector in the predicted joint information sequence, the joints of the electronic device are controlled to move to grasp the target object.

[0012] This application provides a control device, the device comprising:

[0013] The first acquisition module is used to acquire target visual information of the target object and obtain a reference feature vector. The reference feature vector is used to characterize the features of the reference joint angle sequence. The reference joint angle sequence includes the angles of each joint during the process of the electronic device grasping the reference object. The object type of the reference object is the same as the object type of the target object.

[0014] The first prediction module is used to predict the joint information of the electronic device based on the reference feature vector and the target visual information to obtain a predicted joint information sequence. The predicted joint information sequence includes multiple predicted joint vectors, and the predicted joint vectors include the prediction results of each joint in the electronic device.

[0015] The first control module is used to control the movement of each joint of the electronic device based on each predicted joint vector in the predicted joint information sequence, so as to grasp the target object.

[0016] This application provides a computer device, including:

[0017] Memory is used to store executable instructions for a computer;

[0018] The processor, when executing computer-executable instructions stored in the memory, implements the control method provided in the embodiments of this application.

[0019] This application provides a computer-readable storage medium storing computer-executable instructions for implementing the control method provided in this application when executed by a processor.

[0020] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the control method provided in this application. Beneficial effects

[0021] The embodiments of this application have the following beneficial effects:

[0022] In this embodiment, when predicting the joint information of an electronic device, it is based on a reference feature vector and target visual information. The reference feature vector is used to characterize the features of a reference joint angle sequence, which includes the angles of each joint during the electronic device's grasping of a reference object. This reference joint angle sequence can be considered as known grasping example data, thus the reference feature vector can characterize known grasping features. Based on this, when predicting the collected target visual information, the reference feature vector can provide background information to guide the prediction process, thereby improving the accuracy of the predicted joint information sequence. Furthermore, the reference feature vector can also be used as a reference condition, enabling prediction of target objects other than the reference object, thereby improving the generalization ability of the prediction. Moreover, this embodiment predicts multiple predicted joint vectors, not a single predicted joint vector. Therefore, based on each predicted joint vector in the predicted joint information sequence, the movement of each joint can be controlled, enabling fine-grained control of each joint, thereby improving the feasibility, accuracy, and success rate of grasping the target object, and ultimately increasing the efficiency of object grasping. Attached Figure Description

[0023] Figure 1 is a schematic diagram of a network architecture of a control system 100 provided in an embodiment of this application;

[0024] Figure 2 is a structural schematic diagram of the terminal 400 provided in an embodiment of this application;

[0025] Figure 3 is a first flowchart of the control method provided in an embodiment of this application;

[0026] Figure 4 is a second flowchart of the control method provided in an embodiment of this application;

[0027] Figure 5 is a schematic diagram of a process for constructing encoded samples and predicted samples provided in an embodiment of this application;

[0028] Figure 6 is a schematic diagram of the first process for controlling joint movement provided in an embodiment of this application;

[0029] Figure 7 is a schematic diagram of the second process for controlling joint movement provided in an embodiment of this application;

[0030] Figure 8 is a third flowchart of the control method provided in the embodiments of this application;

[0031] Figure 9 is a schematic diagram of the network structure of the conditional neural process model provided in an embodiment of this application.

[0032] Figure 10A is a schematic diagram of the network structure composition of the encoder network using fully connected residual blocks as an embodiment of this application;

[0033] Figure 10B is a schematic diagram of the network structure composition of a queryer network with fully connected residual blocks as provided in an embodiment of this application.

[0034] Figure 11 is a schematic diagram of an item grabbing process provided in an embodiment of this application.

[0035] It should be noted that the terms "first" and "second" are used only to distinguish different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Embodiments of the present invention

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the scope of this application.

[0041] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0042] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0043] 1) Object grasping: In robotics, object grasping refers to the action of a robot using its grasper (such as a robotic arm, gripper, or suction cup) to grasp an object and perform subsequent operations. The grasping process includes perception and recognition, path planning, selection of grasping strategy, execution of grasping actions, and manipulation and placement.

[0044] 2) Joint angles refer to the angles between two segments of the robot arm (or joints) in a robot joint, used to determine the robot arm's motion posture. The angle of each joint can be changed by rotation, thereby adjusting the robot arm's posture, position, or grasping ability.

[0045] Depending on the type of joint, joint angles can take different forms. For joints capable of rotation around a fixed axis, joint angles are called rotational joint angles, representing the range of rotation, usually expressed in degrees or radians. For translational joints with linear motion, joint angles are called translational joint angles, representing the distance the joint moves along a straight line. For joints capable of multi-axis rotation, joint angles are called ball joint angles, used to describe the range of rotation around multiple axes, typically requiring three angular parameters to represent the complete motion of the joint.

[0046] 3) Visual information refers to image or video data such as environment, objects, color, shape, and depth acquired through visual sensors (such as cameras, webcams, and LiDAR). Visual information plays a crucial role in robotics and computer vision, enabling machines or systems to "see" and understand the external environment, thereby achieving tasks such as object recognition, navigation, and grasping.

[0047] Visual information can be acquired through image sensors such as stereo cameras, depth cameras, LiDAR, and ultrasonic sensors. The processing of visual information includes image preprocessing, feature extraction, object recognition, depth estimation and 3D modeling, and motion tracking.

[0048] 4) Time alignment refers to using timestamps to synchronize or match data from different time periods across multiple data sources or events, ensuring that multiple data sources are processed or analyzed under the same time base.

[0049] To better understand the control method provided in the embodiments of this application, the control methods in related technologies and their shortcomings will be explained first.

[0050] Humanoid robots equipped with multi-fingered hands can perform many complex and precise object manipulations, with object grasping being one such operation. However, in related technologies, the object grasping process of humanoid robots with multi-fingered hands faces challenges in modeling and optimization. For example, Gaussian processes, when learning from data, can lead to difficulties in solving problems and high computational complexity. Consequently, when using humanoid robots with multi-fingered hands to grasp objects, issues such as grasping failures and low grasping efficiency arise.

[0051] Based on the above-mentioned technologies, it can be seen that the disadvantages of these technologies include at least the inability to grasp objects, low grasping success rate, and low grasping efficiency.

[0052] This application provides a control method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the accuracy and efficiency of object grasping. The exemplary application of the computer device provided in this application is described below. The computer device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, unmanned vehicles, drones, robots, and aircraft, or it can be implemented as a server. The exemplary application of the computer device as a terminal will be described below.

[0053] Referring to Figure 1, which is a schematic diagram of the network architecture of the control system 100 provided in the embodiment of this application, in order to support an object grasping application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0054] Terminal 400 can be an electronic device such as an unmanned vehicle, drone, robot, or aircraft. Terminal 400 periodically sends feature vector acquisition requests to server 200. Server 200, based on these requests, sends reference feature vectors to terminal 400. These reference feature vectors are pre-determined by server 200 using a reference joint angle sequence, which characterizes the features of that sequence. The reference joint angle sequence includes the angles of each joint during the terminal 400's grasping of the reference object. The object type of the reference object is the same as that of the target object. Terminal 400 also collects target visual information and, based on the reference feature vectors and target visual information, predicts joint information to obtain a predicted joint information sequence. This sequence includes multiple predicted joint vectors, each containing the prediction result for a single joint. Based on each predicted joint vector in the predicted joint information sequence, the terminal controls the movement of each joint to grasp the target object.

[0055] In this embodiment, when predicting joint information, the terminal 400 bases the prediction on a reference feature vector and target visual information. The reference feature vector characterizes the features of a reference joint angle sequence, which includes the angles of each joint during the grasping of the reference object. This reference joint angle sequence can be considered known grasping example data, thus the reference feature vector can characterize known grasping features. Based on this, when predicting the collected target visual information, the reference feature vector provides background information to guide the prediction process, thereby improving the accuracy of the predicted joint information sequence. Furthermore, the reference feature vector can also be used as a reference condition, enabling prediction of target objects other than the reference object, thereby improving the generalization ability of the prediction. Moreover, this embodiment predicts multiple predicted joint vectors, not just a single predicted joint vector. Therefore, each predicted joint vector in the predicted joint information sequence can be used to control the movement of each joint, enabling fine-grained control of each joint, thereby improving the feasibility, accuracy, and success rate of grasping the target object, and ultimately increasing the efficiency of object grasping.

[0056] In some embodiments, the reference feature vector may also be determined in advance by the terminal 400 through a reference joint angle sequence, based on which the terminal 400 can obtain the reference feature vector from itself.

[0057] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0058] Referring to Figure 2, which is a schematic diagram of the structure of a terminal 400 provided in an embodiment of this application, the terminal 400 shown in Figure 2 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440 in Figure 2.

[0059] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0061] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0062] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0063] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0064] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0065] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0066] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0067] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0068] In some embodiments, the apparatus provided in this application can be implemented in software. FIG2 shows a control device 455 stored in memory 450, which can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 4551, a first prediction module 4552, and a first control module 4553. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0069] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the control method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0070] In some embodiments, the terminal can implement the control method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as automatic data capture APPs or automatic item transportation APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0071] The control method provided in this application will be described in conjunction with exemplary applications and implementations of the terminal provided in the embodiments of this application.

[0072] The control method provided in the embodiments of this application will be described below. As mentioned above, the computer device implementing the control method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0073] It should be noted that the control method examples below are illustrated using point cloud data as the target visual information. Those skilled in the art, based on their understanding of the following text, can apply the control method provided in this application to other target visual information such as image data, stereo image pairs, multi-view images, and video sequences. This application can also be applied to various scenarios, including but not limited to industrial automation, logistics and warehousing, medical surgery, home services, agriculture, instant messaging, artificial intelligence, smart transportation, and assisted driving.

[0074] Referring to Figure 3, which is a first flowchart of the control method provided in the embodiment of this application, the control method provided in the embodiment of this application will be described in conjunction with the steps shown in Figure 3. The execution subject in Figure 3 is the terminal.

[0075] In step S101, target visual information of the target object is collected, and a reference feature vector is obtained.

[0076] In this embodiment, the reference feature vector is used to characterize the features of the reference joint angle sequence; the reference joint angle sequence includes the angles of each joint during the process of the electronic device grasping the reference object; the object type of the reference object is the same as the object type of the target object.

[0077] In some embodiments, the target object refers to the object to be grasped by the electronic device. For example, the target object can be a small mechanical part such as a screw, nut, or gear; the target object can also be an assembly such as an automotive part or a circuit board in an electronic product; the target object can also be a packaging such as a box or bottle in the food or pharmaceutical industry; the target object can also be a commodity such as a book, clothing, or toy in an e-commerce warehouse; and the target object can also be a food such as a fruit, vegetable, or meat.

[0078] In some embodiments, a vision sensor is installed on the electronic device to acquire target visual information of a target object. The vision sensor may be at least one of a charge-coupled device camera, a complementary metal-oxide-semiconductor camera, a depth camera, a stereo camera, a laser scanner, a structured light sensor, a 3D vision sensor, a photoelectric sensor, and a tactile sensor. Target visual information refers to visual information including the target object. This target visual information may include point cloud data, and may also include at least one of two-dimensional image data, depth image data, stereo image pairs, optical flow information, semantic labels, contour and edge information, texture features, color histograms, and thermal image data.

[0079] In some embodiments, the reference feature vector is determined in advance based on a reference joint angle sequence, which reflects the characteristics of the reference joint angle sequence. The reference joint angle sequence includes the angles of each joint of the electronic device during the successful grasping of the reference object. For each joint, the reference joint angle sequence includes multiple angles of that joint during the grasping process; that is, the reference joint angle sequence reflects the angle changes of that joint during the grasping process. For example, the reference joint angle sequence can be [(180,180,180), (90,180,180), (60,80,20), (60,80,15)].

[0080] In other embodiments, the initial encoder and initial prediction module can be trained first based on the reference joint angle sequence to obtain a well-trained encoder and a well-trained prediction module. Then, the trained encoder is used to encode the reference joint angle sequence to obtain multiple second initial feature vectors. The mean of each second initial feature vector is then applied to obtain a reference feature vector. This reference feature vector can serve as guidance information, background information, or reference information to guide the prediction of target visual information.

[0081] In some embodiments, the object type of the reference object is the same as the object type of the target object. For example, when the reference object is a bolt, the target object can be one of a bolt, nut, or washer, and in this case, the bolt, nut, and washer are all considered fasteners. When the reference object is a resistor, the target object can be one of a resistor, capacitor, diode, or transistor, and in this case, the resistor, capacitor, diode, and transistor are all considered electronic components.

[0082] In some embodiments, before obtaining the reference feature vector, it is also necessary to determine the reference feature vector. Figure 4 is a second flowchart of the control method provided in the embodiment of this application. That is, before performing the above step S101, referring to Figure 4, the following steps S001 to S007 can also be performed, which will be described in detail below.

[0083] In step S001, multiple sets of reference joint angle sequences of the electronic device are acquired, along with reference visual information corresponding to each reference joint vector in each set of reference joint angle sequences.

[0084] In some embodiments, the electronic device can be controlled to successfully grasp a reference object through remote operation or drag-and-drop teaching. During the process of the electronic device grasping the reference object, the angles of each joint can be periodically collected by the joint position sensor of the electronic device, and reference visual information about the reference object can be periodically collected by the vision sensor. In this embodiment, the acquisition period of the joint position sensor and the vision sensor is the same. The joint position sensor may include at least one of a rotary encoder, potentiometer, gyroscope, magnetic encoder, and optical encoder.

[0085] Here, an angle of each joint is acquired in each acquisition cycle, and a reference joint vector is constructed using these angles. Reference visual information is also acquired, and since the reference joint vector and reference visual information are acquired simultaneously, they can be considered to correspond. Furthermore, each acquisition cycle corresponds to one reference joint vector, and all reference joint vectors constitute a sequence of reference joint angles.

[0086] In step S002, multiple coded samples and multiple predicted samples are constructed based on multiple sets of reference joint angle sequences and various reference visual information.

[0087] In this embodiment, to ensure data length consistency, multiple reference joint angle sequences are first aligned to obtain aligned reference joint angle sequences, with each set of aligned reference joint angle sequences containing the same number of data frames. Then, encoded samples are constructed based on the aligned reference joint angle sequences and corresponding reference visual information. Furthermore, prediction samples are also constructed based on the aligned reference joint angle sequences. The encoded samples are used to train and optimize the first parameters of the initial encoder and the second parameters of the initial prediction module, resulting in a trained encoder and a trained prediction module. The prediction samples are used to evaluate the generalization ability of the trained encoder and the trained prediction module, and also to guide the adjustment of hyperparameters. The encoded samples and prediction samples originate from different aligned reference joint angle sequences.

[0088] In some embodiments, referring to FIG5, the above step S002 can be implemented by the following steps S021 to S026, which are described in detail below.

[0089] In step S021, multiple sets of reference joint angle sequences are time-aligned according to a first preset value T to obtain multiple sets of aligned reference joint angle sequences.

[0090] In this embodiment, each aligned reference joint angle sequence includes T reference joint vectors. The first preset value T is a value set in advance based on experience. For example, for a simple grasping process, the first preset value T can range from 90 to 150; for a complex grasping process, the first preset value T can range from 1000 to 1500.

[0091] In some embodiments, for each set of reference joint angle sequences, if the number of data frames included in the reference joint angle sequence is equal to a first preset value T, it indicates that the reference joint angle sequence does not require alignment processing, and the reference joint angle sequence is determined as an aligned reference joint angle sequence. If the number of data frames included in the reference joint angle sequence is not equal to the first preset value T, then time alignment processing is required for the reference joint angle sequence. If the number of data frames included in the reference joint angle sequence is greater than the first preset value T, then a truncation technique is used for time alignment to obtain an aligned reference joint angle sequence; and if the number of data frames included in the reference joint angle sequence is less than the first preset value T, then an interpolation technique is used for time alignment to obtain an aligned reference joint angle sequence.

[0092] In some embodiments, truncation techniques are used to remove redundant frame data from the reference joint angle sequence. Truncation techniques include start or end truncation and window truncation. By using truncation techniques, the length of the reference joint angle sequence can be shortened, thereby ensuring that the reference joint angle sequence maintains a consistent time range during the analysis process and avoiding unnecessary noise or delayed data.

[0093] In some embodiments, interpolation techniques are used to interpolate data at different times so that each aligned reference joint angle sequence is aligned at the same time point. Common interpolation methods include linear interpolation, polynomial interpolation, spline interpolation, nearest neighbor difference, and time-weighted interpolation.

[0094] In step S022, for each set of aligned reference joint angle sequences, the normalization factor of each reference joint vector in the aligned reference joint angle sequence is determined.

[0095] In this embodiment of the application, for each set of aligned reference joint angle sequences, the aligned reference joint angle sequence includes T reference joint vectors. For each reference joint vector, the normalization factor of the reference joint vector can be determined by the position identifier value of the reference joint vector and the first preset value T, wherein the position identifier value is used to characterize or indicate the position of the reference joint vector in the aligned reference joint angle sequence.

[0096] In some embodiments, the normalization factor of the reference joint vector can also be determined directly by referring to the position identifier value of the joint vector.

[0097] In some embodiments, step S022 can be implemented by steps S221 to S222, which are described in detail below.

[0098] In step S221, the position identifier value of the reference joint vector is obtained.

[0099] In this embodiment of the application, the position identifier value is used to characterize the position of the reference joint vector in the aligned reference joint angle sequence, and the position identifier value is a positive integer less than or equal to T.

[0100] In some embodiments, assuming the reference joint angle sequence includes 100 reference joint vectors, then 1 to 100 can be sequentially determined as the position identifier value of each reference joint vector, so that each reference joint vector has a position identifier value. For example, assuming the reference joint vector is the 50th reference joint vector in the reference joint angle sequence, then the position identifier value of this reference joint vector is 50.

[0101] In step S222, the ratio of the position identifier value to T is determined as the normalization factor of the reference joint vector.

[0102] Continuing with the example above, assuming the position identifier value is 50 and the first preset value T is 100, then 0.5 is determined as the normalization factor for this reference joint vector.

[0103] Through the above steps S221 to S222, a unique normalization factor for each reference joint vector can be determined using the position identifier value and the first preset value T, thereby improving the accuracy and efficiency of the normalization factor. In this way, the position of the reference joint vector in the entire reference joint angle sequence can be quickly and accurately identified based on the normalization factor. Thus, even when a portion of the reference joint vectors is selected from the reference joint angle sequence for analysis, the position of the reference joint vector in the reference joint angle sequence can still be reflected based on the normalization factor, thereby ensuring the sequentiality of the reference joint vectors.

[0104] Please refer to Figure 5 for further explanation following step S022 above.

[0105] In step S023, the normalization factor, the reference joint vector, and the reference visual information corresponding to the reference joint vector are combined to obtain T combined information.

[0106] In some embodiments, the normalization factor, reference joint vector, and corresponding reference visual information can be concatenated in the order of the normalization factor, reference joint vector, and corresponding reference visual information to obtain a combined information. Since each set of aligned reference joint angle sequences includes T reference joint vectors, T combined information can be obtained for each set of aligned reference joint angle sequences.

[0107] In some embodiments, the reference joint vector, reference visual information, and normalization factor can be concatenated in the order of reference joint vector, corresponding reference visual information, and normalization factor to obtain T combined information. Alternatively, the normalization factor, reference visual information, and reference joint vector can be concatenated in the order of normalization factor, corresponding reference visual information, and reference joint vector to obtain T combined information.

[0108] In step S024, any M combined information is determined as M coded samples.

[0109] In this embodiment, M is a positive integer less than T. M is a value set in advance based on experience; for example, when T is 100, M can be 40, 50, 60, etc.

[0110] In some embodiments, M combined information items can be randomly selected from the combined information, and these M combined information items can be determined as M coded samples. The M combined information items can be randomly selected from the combined information using simple random sampling or random subsampling.

[0111] In step S025, the target combined information is obtained from the remaining information set.

[0112] In this embodiment, the remaining information set includes the combined information of the T combined information, excluding the M combined information. That is, the remaining information set does not include any coded sample.

[0113] In some embodiments, at least one target combined information may be obtained from the remaining information set.

[0114] In step S026, the normalization factor and reference visual information in the combined target information are determined as prediction samples.

[0115] In some embodiments, for each target combination information, the normalization factor and reference visual information in the target combination information are concatenated to obtain the predicted sample. Furthermore, the reference joint vector in the target combination information can be determined as the angle vector label of the predicted sample.

[0116] Through steps S021 to S026 above, the reference joint angle sequence is processed into an aligned reference joint angle sequence including T reference joint vectors through time alignment processing, which unifies the number of data frames in the reference joint angle sequence and improves the efficiency of subsequent data processing. Furthermore, based on the reference joint vectors, the normalization factor of the reference joint vectors, and the reference visual information, the combined information is determined, and the encoded sample and the predicted sample carrying the normalization factor are generated based on the combined information, thus ensuring that the encoded sample and the predicted sample have sequence characteristics.

[0117] Please refer to Figure 4 for further explanation following step S002 above.

[0118] In step S003, each encoded sample is encoded based on the initial encoder to obtain the first initial feature vector of each encoded sample, and the mean feature vector is determined based on each first initial feature vector.

[0119] In some embodiments, features can be extracted from each encoded sample to obtain a first initial feature vector for each encoded sample. Each first initial feature vector can characterize the local information of the encoded sample. Then, the first initial feature vectors are averaged to obtain a mean feature vector. This mean feature vector can be considered as a latent representation. Since the mean feature vector retains the global feature information of the encoded sample, it can be considered as the global representation of the encoded sample.

[0120] In step S004, the initial prediction module uses the mean feature vector and prediction samples to perform prediction processing on the joint information to obtain the predicted joint angle vector.

[0121] In this embodiment, the mean feature vector can be considered a global representation, and the mean feature vector serves as a condition to guide the initial prediction module in making predictions.

[0122] In some embodiments, the mean feature vector and the predicted samples are first combined into a predicted feature vector. Since the predicted samples include a normalization factor, the combined predicted feature vector also includes a normalization factor. The predicted feature vector is then input to an initial prediction module to perform prediction processing, thereby obtaining a predicted joint angle vector. The initial prediction module can be a neural network. Combining the mean feature vector and the predicted samples into a predicted feature vector can be achieved by concatenating the mean feature vector and the predicted samples. In some embodiments, a predicted angle variance vector can also be obtained.

[0123] In step S005, the initial encoder and the initial prediction module are trained based on the predicted joint angle vector and the angle vector label of the predicted sample to obtain the trained encoder and the trained prediction module.

[0124] In some embodiments, a loss value can be determined first based on the predicted joint angle vector and the angle vector label. This loss value is then backpropagated to adjust the first parameter of the initial encoder and the second parameter of the initial prediction module, thereby training the initial encoder and the initial prediction module until the training termination condition is met, resulting in a trained encoder and a trained prediction module. The training termination condition can be at least one of the following: the number of training iterations reaches a threshold, or the loss value is less than a threshold. The threshold for the number of training iterations and the loss threshold are values ​​pre-set based on experience. For example, the threshold for the number of training iterations can be 500, 1000, 2000, etc., and the loss threshold can be 0.01, 0.02, 0.03, etc. The trained encoder can be called a trained encoder network. This trained encoder can extract useful information from the encoded samples, generating more accurate reference feature vectors that accurately reflect the patterns and rules behind the encoded samples. The trained prediction module can be a trained decoder network or a trained query network. This trained prediction module can use the reference feature vectors for accurate prediction.

[0125] In step S006, each encoded sample is encoded based on the trained encoder to obtain the second initial feature vector of each encoded sample.

[0126] In this embodiment of the application, the implementation process of step S006 is similar to the implementation process of "encoding each encoded sample based on the initial encoder to obtain the first initial feature vector of each encoded sample" in step S003 above. The difference is that step S003 uses the initial encoder, while step S006 uses a trained encoder. Therefore, the implementation process of step S006 can refer to the implementation process of "encoding each encoded sample based on the initial encoder to obtain the first initial feature vector of each encoded sample" in step S003 above.

[0127] In step S007, the mean of each second initial feature vector is processed to obtain a reference feature vector.

[0128] In this embodiment, all second initial feature vectors can be averaged element-wise to obtain a reference feature vector. That is, the various second initial feature vectors are aggregated into a single reference feature vector through an averaging operation. This reference feature vector is a composite vector or a global feature. For the process of capturing the reference object, this reference feature vector can reflect the successful capture of the reference object.

[0129] Through steps S001 to S007, encoded samples and predicted samples with sequence characteristics are first constructed using reference joint angle sequences. Then, an initial encoder and an initial prediction module are trained based on the encoded and predicted samples, thereby obtaining a trained prediction module with sequence prediction capabilities. Furthermore, a reference feature vector representing global features can be obtained based on the trained encoder, thereby guiding the trained prediction module in predicting target visual information.

[0130] Please refer to Figure 3 for a continuation of step S101 above.

[0131] In step S102, the joint information of the electronic device is predicted based on the reference feature vector and the target visual information to obtain the predicted joint information sequence.

[0132] In the embodiments of this application, the predicted joint information sequence includes multiple predicted joint vectors, and the predicted joint vectors include the prediction results of each joint in the electronic device.

[0133] In some embodiments, the predicted joint information sequence may include a first preset value of T predicted joint vectors. That is, the T predicted results for each joint are obtained in this application embodiment. The T predicted results can form a sequence, thereby realizing the sequence prediction for each joint, so as to guide the joint to perform a series of actions based on the T predicted results, and guide each joint to perform refined actions based on the prediction results of each joint.

[0134] In some embodiments, step S102 can be implemented by steps S1021 and S1022, which are described in detail below.

[0135] In step S1021, the trained prediction module is obtained.

[0136] In this embodiment, the trained prediction module is obtained by training the initial prediction module using a reference joint angle sequence.

[0137] In some embodiments, the trained prediction module is the trained prediction module obtained in step S005 above.

[0138] In step S1022, the trained prediction module is used to predict the joint information of the electronic device based on the reference feature vector and the target visual information, so as to obtain the predicted joint information sequence.

[0139] In this embodiment, for each normalization factor, the normalization factor, reference feature vector, and target visual information can be concatenated to obtain a feature vector to be predicted. This feature vector is then input into a trained prediction module to perform prediction processing, thereby obtaining a predicted joint vector. Since there are a first preset value T normalization factors, a first preset value T feature vectors to be predicted can be obtained, which in turn yields a first preset value T predicted joint vectors. Finally, based on the normalization factors in ascending order, the predicted joint vectors are concatenated to obtain a sequence of predicted joint information. For example, assuming the first preset value T is 100, the normalization factors are 1 / 100, 2 / 100, 3 / 100, ..., 99 / 100, 100 / 100.

[0140] In some embodiments, step S1022 can be achieved through steps S231 and S233, which are described in detail below.

[0141] In step S231, a normalized sequence is constructed based on the first preset value T.

[0142] In this embodiment of the application, the normalized sequence includes T normalization factors.

[0143] Following the example above, the constructed normalized sequence is [1 / 100, 2 / 100, 3 / 100, ..., 99 / 100, 100 / 100], which includes 100 normalization factors.

[0144] In some embodiments, if the first preset value T is 1000, the constructed normalized sequence is [1 / 1000, 2 / 1000, 3 / 1000, ..., 999 / 1000, 1000 / 1000], which includes 1000 normalization factors.

[0145] In step S232, for the k-th normalization factor in the normalization sequence, the trained prediction module is used to predict the joint information of the electronic device based on the k-th normalization factor, target visual information and reference feature vector, to obtain the k-th predicted joint vector.

[0146] In the embodiments of this application, k = 1, 2, 3, ... T.

[0147] In some embodiments, the k-th normalization factor, target visual information, and reference feature vector can be combined to form a feature vector to be predicted; then, the feature vector to be predicted is input into the trained prediction module to obtain the k-th predicted joint vector. The k-th predicted joint vector may include the mean of the predicted angles of each joint, and may also include the variance of the predicted angles of each joint.

[0148] In step S233, the first predicted joint vector to the Tth predicted joint vector are determined as the predicted joint information sequence.

[0149] In this embodiment of the application, the first predicted joint vector to the Tth predicted joint vector are combined sequentially to obtain the predicted joint information sequence.

[0150] Through steps S231 to S233, a normalized sequence is first constructed, which includes T normalization factors. Then, for each basis, normalization factor, target visual information, and reference feature vector, multiple feature vectors to be predicted are constructed. Subsequently, for multiple feature vectors to be predicted, a predicted joint information sequence is predicted. This allows for refined grasping actions of electronic devices based on the predicted joint information sequence.

[0151] Through steps S1021 and S1022, the trained prediction module possesses the ability to predict sequence information and can construct T feature vectors to be predicted based on the k-th normalization factor, target visual information, and reference feature vector. Based on this, the trained prediction module can predict T joint vectors from the T feature vectors to be predicted, thus obtaining a sequence of predicted joint information. This achieves accurate prediction of the grasping process. Furthermore, since the predicted joint information sequence can be output all at once, the prediction efficiency is improved.

[0152] Please refer to Figure 3 for a further explanation following step S102 above.

[0153] In step S103, based on each predicted joint vector in the predicted joint information sequence, the joints of the electronic device are controlled to move to grasp the target object.

[0154] In this embodiment, each predicted joint vector is sequentially obtained from the predicted joint information sequence, and then used as target joint information to control the movement of each joint of the electronic device. Specifically, the first predicted joint vector is used as the first target information to control the movement of each joint of the electronic device until the error between the first actual joint vector and the first predicted joint vector of each joint is less than an error threshold. Then, the second predicted joint vector is used as the second target information to control the movement of each joint of the electronic device until the error between the first actual joint vector and the second predicted joint vector of each joint is less than an error threshold. This process is repeated until the Tth predicted joint vector is used as the Tth target information to control the movement of each joint of the electronic device until the error between the first actual joint vector and the Tth predicted joint vector of each joint is less than an error threshold. This achieves the grasping of the target object. The error threshold is a value pre-set based on experience; for example, the error threshold can be 0.25, 0.20, or 0.15.

[0155] In some embodiments, step S103 can be implemented by steps S1031 and S1032, which are described in detail below.

[0156] In step S1031, the i-th predicted joint vector is obtained from the predicted joint information sequence.

[0157] In the embodiments of this application, i = 1, 2, ... T.

[0158] In some embodiments, the first predicted joint vector, the second predicted joint vector, and so on up to the Tth predicted joint vector are obtained sequentially from the predicted joint information sequence.

[0159] In step S1032, based on the i-th predicted joint vector of each joint of the electronic device, each joint of the electronic device is controlled to move until the error value between the first actual joint vector and the first predicted joint vector is less than the error threshold.

[0160] In this embodiment of the application, the first actual joint vector includes the actual angles of each joint.

[0161] In some embodiments, the i-th predicted joint vector is used as the i-th target information, that is, the i-th predicted joint vector is input as the target value to the control system or controller, so as to control the movement of each joint using the control system or controller. Furthermore, the first actual joint vector of the electronic device can be acquired in real time. During the control process of the control system or controller, at least one of the following—mean square error, mean absolute error, root mean square error, and error vector magnitude—between the first actual joint vector and the first predicted joint vector is determined as the error value between the first actual joint vector and the first predicted joint vector. When the error value is less than the error threshold, the current control is considered complete, and the next predicted joint vector is acquired and control continues based on the next predicted joint vector until the movement of each joint of the electronic device is controlled based on the T-th predicted joint vector. The control system or controller can be at least one of the central processing unit, sensor interface, and driver interface of the electronic device.

[0162] In some embodiments, the step S1032 described above, "controlling the movement of each joint of the electronic device based on the i-th predicted joint vector of each joint of the electronic device", can be implemented in the following two ways, which are described in detail below.

[0163] In the first implementation method, the predicted joint vector includes the average predicted angle of each joint. As shown in Figure 6, the "controlling the movement of each joint of the electronic device based on the i-th predicted joint vector of each joint of the electronic device" in step S1032 can be achieved through steps S321A to S324A, which will be explained in detail below.

[0164] In step S321A, for each joint, the first joint angle and the first velocity information of the joint are collected, and the preset first proportional coefficient and the preset differential coefficient are obtained.

[0165] In some embodiments, the first joint angle and the first velocity information of the joint can be collected in real time by a joint sensor, wherein the first joint angle can refer to the real-time angle of the joint and the first velocity information can refer to the real-time velocity of the joint.

[0166] In some embodiments, the preset first proportional coefficient can be a proportional gain, and the preset derivative coefficient can be a derivative gain. The preset first proportional coefficient and the preset derivative coefficient are pre-set and can be determined through manual adjustment or experimentation. By adjusting the preset first proportional coefficient and the preset derivative coefficient, the response speed and system stability are balanced, preventing system oscillation or overshoot. Once determined, the preset first proportional coefficient and the preset derivative coefficient remain unchanged during the control process.

[0167] In step S322A, the first angle difference is determined based on the first joint angle and the average of the i-th predicted angle of the joint.

[0168] In this embodiment of the application, the difference between the first joint angle and the average value of the i-th predicted angle of the joint is determined as the first angle difference, that is, the difference between the above two angles is determined as the first angle difference.

[0169] In step S323A, first control information is determined based on the first proportional coefficient, the first angle difference, the differential coefficient, and the first velocity information.

[0170] In some embodiments, the first control information may be determined based on the following formula (1): u1(t)=K1·e1(t)+K2·a(t) Formula (1);

[0171] In formula (1), u1(t) refers to the first control information, K1 refers to the first proportional coefficient, e1(t) refers to the first angle difference, K2 refers to the differential coefficient, and a(t) refers to the first velocity information.

[0172] In step S324A, the joint is controlled to move based on the first control information.

[0173] In some embodiments, the first control signal can be the torque or joint speed that the joint needs to be subjected to. When the first control signal is torque, it can directly act on the torque input of the joint motor. The motor in the electronic device applies a corresponding torque according to the received first control signal, pushing the joint to rotate until the average value of the i-th predicted angle is reached. When the first control signal is joint speed, the motor drives the joint to move according to the joint speed until the first angle difference is reduced to zero.

[0174] Through the above steps S321A to S324A, when the predicted joint vector includes the average predicted angle of each joint, the proportional-differential controller can output the first control information. At this time, the predicted joint information sequence is input to the proportional-differential controller, that is, T predicted joint vectors are input to the proportional-differential controller to obtain T first control information. Based on each first control information, the joints are controlled to move in sequence so as to realize the mechanical gripper or clamp of the electronic device to grasp the target object.

[0175] In the second implementation method, the predicted joint vector can include the mean and variance of the predicted angles of each joint. As shown in Figure 7, the "controlling the movement of each joint of the electronic device based on the i-th predicted joint vector of each joint of the electronic device" in step S1032 can be implemented through steps S321B to S324B, which will be explained in detail below.

[0176] In step S321B, for each joint, the second joint angle of the joint is collected, and a preset second proportional coefficient and a preset adjustment factor are obtained.

[0177] In some embodiments, the second joint angle of the joint can be acquired in real time by a joint sensor, wherein the second joint angle can refer to the real-time angle of the joint.

[0178] In some embodiments, the preset second proportional coefficient can be a proportional gain, and the preset adjustment factor can be an adjustment gain. Both the preset second proportional coefficient and the preset adjustment factor are pre-set, and the preset adjustment factor is used to control the sensitivity to angle variance. In complex systems, adaptive algorithms can be used to dynamically adjust the preset adjustment factor according to changes in the environment. In some cases, the preset adjustment factor is adjusted through pre-setting or experience.

[0179] In step S322B, the second angle difference is determined based on the second joint angle and the average of the i-th predicted angle of the joint.

[0180] In some embodiments, the implementation process of step S322B is similar to that of step S322A described above. Therefore, the implementation process of step S322B can refer to the implementation process of step S322A described above.

[0181] In step S323B, the second control information is determined based on the second proportional coefficient, the second angle difference, the adjustment factor, and the variance of the i-th predicted angle of the joint.

[0182] In some embodiments, the second control information can be determined based on the following formula (2): u2(t)=K3·e2(t)+λ·σ 2 (t) Formula (2);

[0183] In formula (2), u2(t) refers to the second control information, K3 refers to the second proportional coefficient, e2(t) refers to the second angle difference, λ refers to the adjustment factor, and σ 2 (t) refers to the variance of the i-th prediction angle.

[0184] In step S324B, the joint is controlled to move based on the second control information.

[0185] In some embodiments, the implementation process of step S324B is similar to that of step S324A described above. Therefore, the implementation process of step S324B can refer to the implementation process of step S324A described above.

[0186] Through the above steps S321B to S324B, when the predicted joint vector includes the mean and variance of the predicted angles of each joint, a robust controller can output second control information. At this time, the predicted joint information sequence is input to the robust controller, that is, T predicted joint vectors are input to the robust controller to obtain T second control information. Based on each second control information, the joints are controlled to move in sequence so as to realize the mechanical gripper or clamp of the electronic device to grasp the target object.

[0187] Through the above steps S1031 and S1032, the movement of each joint of the electronic device can be controlled sequentially based on T predicted joint vectors, thereby realizing process control of each joint in the electronic device and guiding the electronic device to grasp the target object through a series of actions, thus improving the feasibility and success rate of grasping.

[0188] Through steps S101 to S103, the prediction of joint information is based on a reference feature vector and target visual information. The reference feature vector characterizes the features of a reference joint angle sequence, which includes the angles of each joint during the grasping of the reference object. This reference joint angle sequence can be considered as known grasping example data, thus the reference feature vector can characterize known grasping features. Furthermore, a trained prediction module with sequence prediction capabilities can be obtained. Based on this, when using the trained prediction module to predict target visual information, the reference feature vector provides background information to guide the prediction process, thereby improving the accuracy of the predicted joint information sequence. Additionally, a predicted joint information sequence including multiple predicted joint vectors can be obtained. Moreover, the reference feature vector can be used as a reference condition, enabling prediction of target objects other than the reference object, thereby improving the generalization ability of the prediction. Furthermore, the embodiments of this application predict multiple predicted joint vectors, rather than a single predicted joint vector. This allows for the control of each joint's movement based on each predicted joint vector in the predicted joint information sequence, enabling fine-grained control of each joint. This improves the feasibility, accuracy, and success rate of grasping the target object, and ultimately increases the efficiency of object grasping.

[0189] In some embodiments, after the target object is grasped, the electronic device can be controlled to move to the target location and place the target object in a preset position. Referring to Figure 8, after step S103 above, steps S104 to S107 can be performed, which will be described in detail below.

[0190] In step S104, the target location and the current location of the electronic device are obtained.

[0191] In some embodiments, the target location may be a pre-preset location, which may be represented by a two-dimensional coordinate or a three-dimensional coordinate. For example, the target location may be (20, 50) or (20, 50, 15).

[0192] In some embodiments, the current position refers to the real-time position of the electronic device, which can be acquired by a position sensor. The representation of the current position is consistent with the representation of the target position. That is, when the target position is represented by two-dimensional coordinates, the current position is also represented by two-dimensional coordinates, and the target position and the current position are referenced to the same reference coordinate system.

[0193] In step S105, path planning is performed based on the target location and the current location to obtain the movement path of the electronic device.

[0194] In some embodiments, environmental information may also be obtained. A path planning model may be used to predict the path based on the target location, the current location, and the environmental information to obtain a movement path. This movement path is used to indicate that the electronic device moves from the current location to the target location, and there are no obstacles on the movement path.

[0195] In step S106, the electronic device is controlled to move based on the movement path until the target position is reached.

[0196] In some embodiments, the movement path may consist of multiple path points, and the electronic device may be controlled to move to each path point in sequence according to the order of the path points until the last path point, thereby controlling the electronic device to move to the target location.

[0197] In step S107, the electronic device is controlled to perform a release action, placing the target object at a preset position.

[0198] In some embodiments, the release action may be at least one of the opening of a mechanical gripper or clamp, or the de-energization of an electromagnet, with the purpose of allowing the electronic device to release the target object it has grasped. The preset position is a pre-set location, or it may be determined in real-time by the electronic device via a vision sensor during movement. This preset position is used to place the target object; for example, the preset position may be the location of a plate on a dining table, or it may be the next workstation on a production line, a designated location on a shelf, or a packaging area, etc.

[0199] In some embodiments, steps S104 and S105 may be executed before step S101, or step S101 may be executed simultaneously. This application embodiment does not limit the execution order of steps S104, S105, and S101. Steps S106 and S107 are executed after step S103; that is, the target object is first grasped, then moved to the target location, and finally placed at the preset location.

[0200] Through the above steps S104 to S107, during or after the electronic device grasps the target object, the movement path of the electronic device can be determined. After the electronic device grasps the target object, it is controlled to move to the target position based on the movement path, and then the electronic device is controlled to perform a release action to place the target object in the preset position, thereby achieving the purpose of automatically moving the target object and improving the efficiency of moving the target object.

[0201] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0202] This application proposes a control method for a humanoid robot object grasping process based on conditional neural processes. Through teleoperation or drag-and-drop teaching functions, teaching data is collected during the humanoid robot's object grasping process. The collected data is then fed into a conditional neural process network for learning. The learned conditional neural process model is used to achieve object grasping by the humanoid robot. Here, the humanoid robot corresponds to the electronic device in other embodiments, the object corresponds to the reference object in other embodiments, and the collected data corresponds to the reference joint angle sequence and reference visual information in other embodiments.

[0203] In some embodiments, humanoid robots equipped with multi-fingered hands can perform many complex and precise object manipulations, with object grasping being one such operation. The object grasping process for humanoid robots equipped with multi-fingered hands presents challenges in modeling and optimization. Deep imitation learning methods allow humanoid robots to directly learn the data association between multi-fingered hand movements and object states during object grasping from teaching data. Furthermore, considering that object grasping is a process, i.e., the teaching data is a sequence of actions, such as opening the multi-fingered hand, approaching the object, and grasping the object, a method capable of learning the object grasping action sequence is needed. Conditional neural processes, leveraging modern gradient optimization techniques such as error backpropagation in deep learning, avoid the computational difficulties and high complexity associated with learning data using traditional stochastic processes (such as Gaussian processes).

[0204] In this embodiment, to achieve the learning of object grasping using a humanoid robot equipped with a multi-fingered hand, a humanoid robot object grasping learning method based on conditional neural processes is proposed. Multiple sets of data sequences of humanoid robot object grasping are collected through teaching. Each frame of this data sequence contains the position and posture data of the humanoid robot's end-effector wrist, the joint angles of each driven joint of the multi-fingered hand, and visual information of the object, such as point cloud information. After normalizing the sequence length, the method is learned through conditional neural processes to achieve the goal of learning the object grasping process, thereby avoiding the problems of complex modeling and optimization difficulties in the object grasping process. The data sequences correspond to the reference joint angle sequences and reference visual information in other embodiments; the visual information of the object corresponds to the reference visual information in other embodiments; and the position and posture data of the humanoid robot's end-effector wrist and the joint angles of each driven joint of the multi-fingered hand correspond to the reference joint angle sequences in other embodiments.

[0205] The implementation process of this application embodiment mainly includes three stages: a teaching data acquisition stage, a data cleaning and learning stage, and an object grasping stage. The teaching data acquisition stage combines the drag-and-drop teaching function or teleoperation function of the humanoid robot with a visual sensor to collect teaching data. The collected teaching data is then cleaned and organized into a data sequence in a preset format; this preset format data sequence is then input into a conditional neural process model for model learning. The learned model will infer a complete action sequence based on given object inference information during subsequent humanoid robot object grasping. The teaching data corresponds to the reference joint angle sequence and reference visual information in other embodiments, and the preset format data sequence corresponds to the combined information in other embodiments.

[0206] During the teaching data acquisition phase, if the humanoid robot has a drag-and-drop teaching function, this function can be used directly to acquire the robot's motion data, or a teleoperation device can be used to acquire the humanoid robot's motion data. The acquired humanoid robot motion data can include joint information for each degree of freedom of the robot, including the multi-fingered hand, or the position and posture information of the robot's end effector wrist and the joint angles for each degree of freedom of the multi-fingered hand. In addition, a visual sensor (such as a depth camera) is needed to acquire the object's visual information. This visual information can be the object's point cloud information or the object's spatial position information in camera coordinates.

[0207] During the data cleaning process, the teaching data will be organized into the form shown in equation (3).

[0208] In equation (3), z tz represents the normalization factor for the length of the action sequence. t The value of z is t / T. t Corresponding to the normalization factor in other embodiments, where T is the length of the action sequence, T corresponds to the first preset value in other embodiments, t is the sequence number of each frame of data in the action sequence, and t corresponds to the position identifier value in other embodiments; O i,t D represents the visual information of the object at frame t in the i-th set of teaching data. o O i,t Data dimension. Specifically, the point cloud information of objects can be extracted using classic PointNet or PointNet++ networks to reduce O(log n) complexity. i,t The dimension. A i,t Then, D represents the angles of each joint in the robot, including the multi-fingered hand, at frame t in the i-th set of teaching data. A A represents i,t Data dimension. Suppose N sets of teaching data after the above-mentioned tissues are collected, and these data are fed into the conditional neural process model for learning.

[0209] In this embodiment, the network structure of the conditional neural process model is shown in Figure 9. As shown in Figure 9, the network structure includes an encoder network 901, a mean processing module 902, and a queryer network 903. M frames of data d are randomly selected from the training dataset each time. i As input data 904, the input data 904 is fed into the encoder network 901, which determines the representation vector r of each frame of data. i ∈R P The representation vector is a latent variable, and P is a manually set dimension. Next, the mean value processing module 902 determines the mean value r∈R of the representation vectors based on each representation vector. N×P Finally, the mean of the representation vector will be compared with the original training data d. t [z] t O t Together, they are passed into query network 903. Query network 903 will predict the outcome regarding A. t The parameters of the normal distribution, which are also the output information 905. Here, encoder network 901 corresponds to the initial encoder in other embodiments, query network 903 corresponds to the initial prediction module in other embodiments, training dataset corresponds to the combined information in other embodiments, input data 904 corresponds to the encoded samples in other embodiments, representation vector corresponds to the first initial feature vector in other embodiments, and the mean of representation vector corresponds to the mean feature vector in other embodiments. [z t O t ] represents the predicted sample, A tThe parameters of the normal distribution represent the predicted joint angle vector.

[0210] The learning process of the above-mentioned conditional neural process model essentially involves continuously adjusting the parameters of each neuron in the query network 901 and the decoder network 903 based on the training data. Here, various existing deep learning optimizers, such as Adam and SGD, can be directly utilized. The loss function of this conditional neural process model during the learning process is shown in formula (4): L(θ,φ)=-logP(A t |z t O t ) formula (4);

[0211] In formula (4), θ represents the parameters of the encoder network, corresponding to the first parameter in other embodiments; φ represents the parameters of the query network, corresponding to the second parameter in other embodiments; the above formula (3) can be understood as maximizing the conditional probability distribution P(A t |z t O t The likelihood estimation of ) is performed. Finally, a trained encoder network and a trained query network are obtained through training, wherein the trained encoder network corresponds to the trained encoder in other embodiments, and the trained query network corresponds to the trained prediction module in other embodiments.

[0212] The network design within the encoder and query networks can be modified according to the complexity of the object's visual information. For example, if the object's visual information is point cloud information, a multi-level cascaded fully connected residual block can be considered; if the object's visual information is red-green-blue image information, a convolutional neural network can be considered; and if the object's information only includes its position and pose values, a fully connected neural network can be used directly. Figure 10A shows the network structure of the encoder network using fully connected residual blocks, which includes a weight layer 1001, an activation function layer 1002, a weight layer 1003, and a 1x1 convolutional layer 1004. Figure 10B shows the network structure of the query network using fully connected residual blocks, which includes a weight layer 1005, an activation function layer 1006, a weight layer 1007, and a 1x1 convolutional layer 1008.

[0213] During the object grasping phase, the information O obtained from the visual sensor of the object is... t The normalized factor sequence and the mean of the representation vector are input together into the trained queryer network, where the normalized factor sequence is... Corresponding to the normalized sequence in other embodiments, the trained query network corresponds to the trained prediction module in other implementations; based on this, the joint angles of each degree of freedom of the robot, including the multi-fingered hand, can be obtained, denoted as [A1, A2, ..., A...]. T ], which corresponds to the predicted joint information sequence in other embodiments.

[0214] In some embodiments, a proportional-derivative (PD) controller can be used to detect and control joint angles in robot control. The mathematical expression of the PDC controller is shown in formula (5):

[0215] In formula (5), A c This represents the joint angles of the humanoid robot in each degree of freedom. K represents the velocity values ​​of each degree of freedom of the humanoid robot. p K represents the proportionality coefficient. d This represents the differential coefficient.

[0216] Figure 11 is a flowchart of an object grasping process provided in an embodiment of this application. Referring to Figure 11, object grasping can be achieved through the following steps S1101 to S1109, which will be described in detail below.

[0217] In step S1101, the capture is initiated.

[0218] In step S1102, object information and normalized sequence are obtained.

[0219] In this embodiment of the application, the object information can be denoted as O. t The object information corresponds to the target visual information in other embodiments. This normalization sequence includes multiple normalization factors. For example, the normalization sequence can be [1 / T, 2 / T, 3 / T, ..., (T-1) / T, T / T], where T is the length of the action sequence. During the object grasping process, the normalization factors are used to control the generation of T joint angle mean sequences.

[0220] In step S1103, the object information and normalized sequence are input into the trained query network to obtain the mean sequence of joint angles for each degree of freedom.

[0221] In this embodiment, the trained query network corresponds to the trained prediction module in other embodiments. In this embodiment, a reference feature vector is also input into the trained query network; this reference feature vector refers to the mean of the representation vector output by the trained encoder network. The mean sequence of joint angles corresponds to the predicted joint information sequence in other embodiments, corresponding to the aforementioned [A1, A2, ..., A...]. T ].

[0222] In step S1104, the average sequence of joint angles for each degree of freedom is input to the proportional-derivative controller.

[0223] In this embodiment, the sequence of joint angle mean values ​​for each degree of freedom includes T joint angle mean values ​​for each degree of freedom. The joint angle mean values ​​correspond to the predicted joint vectors in other embodiments. The first step may be to input the first joint angle mean value to the proportional-derivative controller, that is, A1 will be input to the proportional-derivative controller.

[0224] In step S1105, the joint angles of each degree of freedom are collected.

[0225] In this embodiment of the application, joint angles for each degree of freedom can be collected using joint sensors, and each joint angle represents the aforementioned A. c .

[0226] In step S1106, it is determined whether the joint angles of each degree of freedom are the same as the average joint angle.

[0227] In this embodiment of the application, the average value of each joint angle for each degree of freedom is obtained sequentially. If the joint angle of each degree of freedom is the same as the average value of the joint angle, it is determined whether it is the last average value of the joint angle, that is, proceed to step S1107; if there is any degree of freedom whose joint angle is not equal to the average value of the joint angle, it returns to step S1104, that is, it continues to control based on proportional-derivative control until the joint angle of each degree of freedom is the same as the average value of the joint angle.

[0228] In step S1107, it is determined whether it is the average angle of the last joint.

[0229] In this embodiment of the application, if it is not the average angle of the last joint, then the average angle of the next joint is obtained, that is, the process proceeds to step S1108; if it is the average angle of the last joint, it indicates that each joint has been controlled based on the average angle of each joint, and the process ends at this time, that is, the process proceeds to step S1109.

[0230] In step S1108, the next average joint angle is obtained. Then, the process returns to step S1103.

[0231] In this embodiment, the next average joint angle is obtained from the average joint angle sequence, i.e., returning to step S1103, and the obtained next average joint angle is input to the proportional-derivative controller. Step S1108 is used to input the average joint angles from the second to the Tth joint angles to the proportional-derivative controller, i.e., A2,…,A T The inputs are sequentially fed into the proportional-derivative controller.

[0232] In step S1109, the process ends.

[0233] Through the above steps S1101 to S1109, a method for grasping objects for humanoid robots equipped with multi-fingered hands is provided, which helps to improve the feasibility of humanoid robots in dexterous operation tasks such as object manipulation.

[0234] It is understood that in the embodiments of this application, data related to target visual information, reference feature vectors, first label information, three-dimensional sample data, second label information, and movement paths are involved. The collection, use, and processing of such data must comply with relevant laws, regulations, and standards.

[0235] The following continues to describe an exemplary structure of the control device 455 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2, the software module stored in the control device 455 in the memory 450 may include:

[0236] The first acquisition module 4551 is used to acquire target visual information of the target object and obtain a reference feature vector. The reference feature vector is used to characterize the features of the reference joint angle sequence. The reference joint angle sequence includes the angles of each joint during the process of the electronic device grasping the reference object. The object type of the reference object is the same as the object type of the target object.

[0237] The first prediction module 4552 is used to predict the joint information of the electronic device based on the reference feature vector and the target visual information to obtain a predicted joint information sequence. The predicted joint information sequence includes multiple predicted joint vectors, and the predicted joint vectors include the prediction results of each joint in the electronic device.

[0238] The first control module 4553 is used to control the movement of each joint of the electronic device based on each predicted joint vector in the predicted joint information sequence, so as to grasp the target object.

[0239] In some embodiments, the first prediction module 4552 is further configured to:

[0240] A trained prediction module is obtained, which is trained on an initial prediction module using the reference joint angle sequence. Using the trained prediction module, the joint information of the electronic device is predicted based on the reference feature vector and the target visual information to obtain the predicted joint information sequence.

[0241] In some embodiments, the first prediction module 4552 is further configured to:

[0242] A normalized sequence is constructed based on a first preset value T, the normalized sequence including T normalization factors; for the k-th normalization factor in the normalized sequence, the trained prediction module is used to predict the joint information of the electronic device based on the k-th normalization factor, the target visual information, and the reference feature vector, to obtain the k-th predicted joint vector, k = 1, 2, 3, ... T; the 1st predicted joint vector to the Tth predicted joint vector are determined as the predicted joint information sequence.

[0243] In some embodiments, the first control module 4553 is further configured to:

[0244] The i-th predicted joint vector is obtained from the predicted joint information sequence, i = 1, 2, ... T; based on the i-th predicted joint vector of each joint of the electronic device, the movement of each joint of the electronic device is controlled until the error value between the first actual joint vector and the first predicted joint vector is less than the error threshold, wherein the first actual joint vector includes the actual angle of each joint.

[0245] In some embodiments, the predicted joint vector includes the average predicted angle of each of the joints, and the first control module 4553 is further configured to:

[0246] For each joint, the first joint angle and first velocity information of the joint are collected, and a preset first proportional coefficient and a preset differential coefficient are obtained; based on the first joint angle and the average value of the i-th predicted angle of the joint, a first angle difference is determined; based on the first proportional coefficient, the first angle difference, the differential coefficient and the first velocity information, first control information is determined; based on the first control information, the joint is controlled to move.

[0247] In some embodiments, the predicted joint vector includes the mean and variance of the predicted angles of each of the joints, and the first control module 4553 is further configured to:

[0248] For each joint, the second joint angle of the joint is collected, and a preset second proportional coefficient and a preset adjustment factor are obtained; based on the second joint angle and the mean of the i-th predicted angle of the joint, a second angle difference is determined; based on the second proportional coefficient, the second angle difference, the adjustment factor and the variance of the i-th predicted angle of the joint, second control information is determined; based on the second control information, the joint is controlled to move.

[0249] In some embodiments, the software module in the control device 455 may further include:

[0250] The second acquisition module is used to acquire multiple sets of reference joint angle sequences of the electronic device, as well as reference visual information corresponding to each reference joint vector in each set of reference joint angle sequences.

[0251] The construction module is used to construct multiple encoded samples and multiple predicted samples based on multiple sets of the reference joint angle sequences and each of the reference visual information;

[0252] The first encoding module is used to encode each of the encoded samples based on the initial encoder to obtain a first initial feature vector for each encoded sample, and to determine a mean feature vector based on each of the first initial feature vectors.

[0253] The second prediction module is used to perform prediction processing on the joint information based on the mean feature vector and the prediction sample by the initial prediction module to obtain the predicted joint angle vector.

[0254] The training module is used to train the initial encoder and the initial prediction module based on the predicted joint angle vector and the angle vector label of the predicted sample, so as to obtain the trained encoder and the trained prediction module.

[0255] In some embodiments, the building module is further configured to:

[0256] Multiple sets of reference joint angle sequences are time-aligned according to a first preset value T to obtain multiple sets of aligned reference joint angle sequences. Each set of aligned reference joint angle sequences includes T reference joint vectors. For each set of aligned reference joint angle sequences, a normalization factor is determined for each reference joint vector in the aligned reference joint angle sequence. The normalization factor, the reference joint vector, and the reference visual information corresponding to the reference joint vector are combined to obtain T combined information. Any M combined information are determined as M encoded samples, where M is a positive integer less than T. Target combined information is obtained from the remaining information set, which includes combined information other than the M combined information from the T combined information. The normalization factor and reference visual information in the target combined information are determined as the predicted samples.

[0257] In some embodiments, the building module is further configured to:

[0258] Obtain the position identifier value of the reference joint vector, which is used to characterize the position of the reference joint vector in the aligned reference joint angle sequence, and the position identifier value is a positive integer less than or equal to T; determine the ratio of the position identifier value to T as the normalization factor of the reference joint vector.

[0259] In some embodiments, the software module in the control device 455 may further include:

[0260] The second encoding module is used to encode each encoded sample based on the trained encoder to obtain a second initial feature vector for each encoded sample.

[0261] The mean processing module is used to perform mean processing on each of the second initial feature vectors to obtain the reference feature vector.

[0262] In some embodiments, the software module in the control device 455 may further include:

[0263] An acquisition module is used to acquire the target location and the current location of the electronic device;

[0264] The planning module is used to perform path planning based on the target location and the current location to obtain the movement path of the electronic device;

[0265] The second control module is used to control the electronic device to move based on the movement path until it reaches the target position;

[0266] The third control module is used to control the electronic device to perform a release action and place the target object in a preset position.

[0267] This application provides a computer program product or computer program that includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the control method described in this application.

[0268] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the control method provided in this application, such as the control method shown in Figures 3, 4, and 8.

[0269] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0270] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0271] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0272] As an example, computer-executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0273] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A control method, characterized in that, The method includes: Visual information of the target object is collected, and a reference feature vector is obtained. The reference feature vector is used to characterize the features of the reference joint angle sequence. The reference joint angle sequence includes the angles of each joint during the process of the electronic device grasping the reference object. The object type of the reference object is the same as the object type of the target object. Based on the reference feature vector and the target visual information, the joint information of the electronic device is predicted to obtain a predicted joint information sequence. The predicted joint information sequence includes multiple predicted joint vectors, and the predicted joint vectors include the prediction results of each joint in the electronic device. Based on each predicted joint vector in the predicted joint information sequence, the joints of the electronic device are controlled to move to grasp the target object.

2. The method according to claim 1, characterized in that, The step of predicting the joint information of the electronic device based on the reference feature vector and the target visual information to obtain a predicted joint information sequence includes: A trained prediction module is obtained, wherein the trained prediction module is trained on the initial prediction module using the reference joint angle sequence; Using the trained prediction module, based on the reference feature vector and the target visual information, the joint information of the electronic device is predicted to obtain the predicted joint information sequence.

3. The method according to claim 2, characterized in that, The step of using the trained prediction module to predict the joint information of the electronic device based on the reference feature vector and the target visual information to obtain the predicted joint information sequence includes: A normalized sequence is constructed based on a first preset value T, wherein the normalized sequence includes T normalization factors; For the k-th normalization factor in the normalization sequence, the trained prediction module is used to predict the joint information of the electronic device based on the k-th normalization factor, the target visual information, and the reference feature vector, to obtain the k-th predicted joint vector, k = 1, 2, 3, ... T; The first to the Tth predicted joint vectors are determined as the predicted joint information sequence.

4. The method according to claim 3, characterized in that, The step of controlling the movement of each joint of the electronic device based on each predicted joint vector in the predicted joint information sequence includes: Obtain the i-th predicted joint vector from the predicted joint information sequence, where i = 1, 2, ..., T; Based on the i-th predicted joint vector of each joint of the electronic device, the movement of each joint of the electronic device is controlled until the error value between the first actual joint vector and the first predicted joint vector is less than the error threshold, wherein the first actual joint vector includes the actual angle of each joint.

5. The method according to claim 4, characterized in that, The predicted joint vector includes the average predicted angle of each joint. Controlling the movement of each joint of the electronic device based on the i-th predicted joint vector of each joint includes: For each joint, the first joint angle and the first velocity information of the joint are collected, and a preset first proportional coefficient and a preset differential coefficient are obtained. The first angle difference is determined based on the first joint angle and the average of the i-th predicted angle of the joint; Based on the first proportional coefficient, the first angle difference, the differential coefficient, and the first velocity information, the first control information is determined; The joint is controlled to move based on the first control information.

6. The method according to claim 4, characterized in that, The predicted joint vector includes the mean and variance of the predicted angles for each joint. Controlling the movement of each joint of the electronic device based on the i-th predicted joint vector of each joint includes: For each joint, the second joint angle of the joint is collected, and a preset second proportional coefficient and a preset adjustment factor are obtained; The second angle difference is determined based on the second joint angle and the average of the i-th predicted angle of the joint; The second control information is determined based on the second proportional coefficient, the second angle difference, the adjustment factor, and the variance of the i-th predicted angle of the joint. The joint is controlled to move based on the second control information.

7. The method according to any one of claims 1 to 6, characterized in that, Before obtaining the reference feature vector, the method further includes: Collect multiple sets of reference joint angle sequences of the electronic device, as well as reference visual information corresponding to each reference joint vector in each set of reference joint angle sequences; Based on multiple sets of the reference joint angle sequences and each of the reference visual information, multiple encoded samples and multiple predicted samples are constructed. The encoding samples are encoded based on the initial encoder to obtain the first initial feature vector of each encoding sample, and the mean feature vector is determined based on each first initial feature vector. The initial prediction module uses the mean feature vector and the prediction sample to perform prediction processing on the joint information to obtain the predicted joint angle vector; The initial encoder and the initial prediction module are trained based on the predicted joint angle vector and the angle vector label of the predicted sample to obtain the trained encoder and the trained prediction module.

8. The method according to claim 7, characterized in that, Based on multiple sets of reference joint angle sequences and various reference visual information, multiple encoded samples and multiple predicted samples are constructed, including: The multiple sets of reference joint angle sequences are time-aligned according to a first preset value T to obtain multiple sets of aligned reference joint angle sequences. Each set of aligned reference joint angle sequences includes T reference joint vectors. For each set of aligned reference joint angle sequences, determine the normalization factor for each reference joint vector in the aligned reference joint angle sequence. By combining the normalization factor, the reference joint vector, and the reference visual information corresponding to the reference joint vector, T combined information items are obtained; Any M combined information is used to determine M encoded samples, where M is a positive integer less than T; Obtain the target combined information from the remaining information set, wherein the remaining information set includes the combined information other than the M combined information from the T combined information sets; The normalization factor and reference visual information in the combined target information are determined as the prediction sample.

9. The method according to claim 8, characterized in that, The step of determining the normalization factor for each reference joint vector in the aligned reference joint angle sequence includes: Obtain the position identifier value of the reference joint vector, the position identifier value is used to characterize the position of the reference joint vector in the aligned reference joint angle sequence, and the position identifier value is a positive integer less than or equal to T; The ratio of the position identifier value to T is determined as the normalization factor of the reference joint vector.

10. The method according to claim 9, characterized in that, The method further includes: Based on the trained encoder, each of the encoded samples is encoded to obtain the second initial feature vector of each encoded sample; The reference feature vector is obtained by averaging each of the second initial feature vectors.

11. The method according to any one of claims 1 to 6, characterized in that, After crawling the target object, the method further includes: Obtain the target location and the current location of the electronic device; Based on the target location and the current location, path planning is performed to obtain the movement path of the electronic device; The electronic device is controlled to move based on the movement path until the target location is reached; The electronic device is controlled to perform a release action, placing the target object at a preset position.

12. A control device, characterized in that, The device includes: The first acquisition module is used to acquire target visual information of the target object and obtain a reference feature vector. The reference feature vector is used to characterize the features of the reference joint angle sequence. The reference joint angle sequence includes the angles of each joint during the process of the electronic device grasping the reference object. The object type of the reference object is the same as the object type of the target object. The first prediction module is used to predict the joint information of the electronic device based on the reference feature vector and the target visual information to obtain a predicted joint information sequence. The predicted joint information sequence includes multiple predicted joint vectors, and the predicted joint vectors include the prediction results of each joint in the electronic device. The first control module is used to control the movement of each joint of the electronic device based on each predicted joint vector in the predicted joint information sequence, so as to grasp the target object.

13. A computer device, characterized in that, The computer device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the control method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the control method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the control method according to any one of claims 1 to 11 is implemented.

Images