Dexterous hand control method and device, and agent and storage medium
By combining cerebellar neural network model with whole-body dynamics model model prediction control, the problem of insufficient precision and flexibility of dexterous hands in high-degree-of-freedom task scenarios is solved, and more efficient control precision and environmental adaptability are achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HUMANOID ROBOT (SHANGHAI) CO LTD
- Filing Date
- 2025-05-23
- Publication Date
- 2026-07-23
AI Technical Summary
Traditional dexterous hand control strategies lack precision and flexibility in high-degree-of-freedom, rapid-response, and highly coordinated task scenarios, making it difficult to adapt to changing operating environments.
By introducing the cerebellum mechanism and combining the cerebellum model neural network with the whole-body dynamics model, trajectory planning and motion control are performed through model predictive control algorithms, thereby improving the control accuracy and flexibility of dexterous hands.
It improves the control precision and flexibility of dexterous hands in complex tasks, enabling them to quickly adapt to different operating environments and task requirements, reduce reliance on precise mathematical models, and simplify control system design.
Smart Images
Figure CN2025097015_23072026_PF_FP_ABST
Abstract
Description
Dexterous hand control methods, devices, intelligent agents, and storage media
[0001] This application claims priority to Chinese Patent Application No. 202510073194.6, filed on January 17, 2025, entitled “Dexterous Hand Control Method, Device, Intelligent Agent and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of humanoid robot dexterous hand technology, and more particularly to a dexterous hand control method, device, intelligent agent, and storage medium. Background Technology
[0003] With the rapid advancement of technology, robotics is developing at an unprecedented pace. Dexterous hands, as a prominent representative of this field, possess remarkable capabilities for complex and precise manipulation. However, this high degree of freedom also brings unprecedented control challenges, especially in task scenarios requiring rapid response and high coordination. Traditional control strategies, while performing well in certain scenarios, are often limited by complex mathematical models and fixed algorithmic frameworks, making it difficult to flexibly adapt to changing operating environments.
[0004] In related technologies, traditional control strategies, such as proportional-integral-differential (PID) control, classical adaptive control, and open-loop control, can be used to control the dexterous hand.
[0005] However, in the process of realizing this application, the inventors discovered that the prior art has at least the following problems: for dexterous hands with high degrees of freedom, the above methods lack precision and flexibility in task scenarios that require rapid response and high coordination. Summary of the Invention
[0006] This application provides a dexterous hand control method, device, intelligent agent, and storage medium to improve the accuracy and flexibility of dexterous hand control.
[0007] In a first aspect, this application provides a method for controlling a dexterous hand, comprising:
[0008] Acquire sensor data;
[0009] The sensor data is preprocessed to obtain input information; the input information includes environmental information and dexterity hand status information.
[0010] The input information is input into the cerebellum model to obtain corresponding control commands; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output corresponding trajectory planning results according to the input information, and the motion controller is used to convert the trajectory planning results into corresponding control commands; the motion controller is obtained by combining the cerebellum model neural network with the whole-body dynamics model based on the model predictive control algorithm;
[0011] The actuator module of the dexterous hand is driven and controlled according to the control command.
[0012] In one possible design, the environmental information includes at least one of the following parameters of an object in the real environment in which the dexterous hand is located: position, shape, size; and the state information includes at least one of the following parameters: position, force, speed, and acceleration of the dexterous hand's fingers.
[0013] In one possible design, before inputting the input information into the cerebellum model to obtain the corresponding control command, the following steps are also included:
[0014] Based on the physical structure of the dexterous hand and the robotic arm, construct a physical model;
[0015] Construct a cerebellar neural network to be trained and a trajectory planning model to be trained;
[0016] Obtain the training sample set;
[0017] Based on the training sample set and the physical model, the cerebellar neural network to be trained and the trajectory planning model to be trained are trained to obtain the cerebellar model neural network and the trajectory planning model;
[0018] The cerebellar model neural network is combined with the whole-body dynamics model based on the model predictive control algorithm to obtain a motion controller, and the cerebellar model is determined according to the motion controller and the trajectory planning model.
[0019] In one possible design, the model predictive control algorithm combines a cerebellar model neural network with a whole-body dynamics model to obtain a motion controller, including:
[0020] The inverse kinematics module and the inverse dynamics module are connected in series and then connected in parallel with the cerebellar model neural network to obtain a motion controller. The inverse dynamics module is also used to connect in series with the forward dynamics module and the forward kinematics module in sequence. The forward dynamics module is used to provide feedback output to the inverse dynamics module, and the forward kinematics module is used to provide feedback output to the inverse kinematics module and the cerebellar model neural network.
[0021] In one possible design, the model predictive control algorithm combines a cerebellar model neural network with a whole-body dynamics model to obtain a motion controller, including:
[0022] A motion controller is obtained by sequentially connecting a cerebellar model neural network with a positive dynamics module and a positive kinematics module; wherein, the positive kinematics module is used to provide feedback output to the cerebellar model neural network.
[0023] In one possible design, the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7.
[0024] In one possible design, obtaining the training sample set includes:
[0025] The system remotely controls a dexterous hand and robotic arm to perform grasping and manipulation tasks in multiple scenarios, and collects sensor data during task execution. The task objects or operating environments are different in different scenarios. Different task objects refer to task objects that have different parameters of at least one of the following: shape, material, or position.
[0026] The sensor data is preprocessed to obtain a training sample set.
[0027] In one possible design, the training sample set includes multiple sets of image sequences and corresponding desired trajectories, as well as multiple sets of trajectories and corresponding desired moments; the step of training the cerebellar neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model includes:
[0028] Based on the first loss function and the backpropagation algorithm, the cerebellar neural network to be trained is trained according to multiple sets of trajectories and corresponding expected torques to obtain the cerebellar model neural network; the first loss function includes the error between the actual torque and the expected torque output by the cerebellar neural network to be trained.
[0029] Based on the second loss function and the backpropagation algorithm, the trajectory planning model is trained according to multiple sets of image sequences and corresponding expected trajectories to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model and the expected trajectory.
[0030] In one possible design, the training sample set includes multiple sets of trajectories and corresponding expected moments, as well as multiple sets of image sequences and corresponding expected moments; the step of training the cerebellar neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model includes:
[0031] Based on the first loss function and the backpropagation algorithm, the cerebellar neural network to be trained is trained according to multiple sets of trajectories and corresponding expected torques to obtain an initial cerebellar model neural network; the first loss function includes the error between the actual torque and the expected torque output by the cerebellar neural network to be trained.
[0032] Based on the second loss function and the backpropagation algorithm, the trajectory planning model to be trained and the initial cerebellar model neural network are jointly trained according to multiple sets of image sequences and corresponding expected moments to obtain the trajectory planning model and the cerebellar model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.
[0033] Secondly, this application provides a dexterous hand control device, comprising:
[0034] The acquisition module is used to acquire sensor data;
[0035] The determination module is used to preprocess the sensor data to obtain input information; the input information includes environmental information and dexterity hand status information.
[0036] The processing module is used to input the input information into the cerebellum model to obtain corresponding control commands; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output corresponding trajectory planning results according to the input information, and the motion controller is used to convert the trajectory planning results into corresponding control commands; the motion controller is obtained by combining the cerebellum model neural network with the whole-body dynamics model based on the model predictive control algorithm.
[0037] The drive control module is used to drive and control the actuator module of the dexterous hand according to the control command.
[0038] Thirdly, this application provides a dexterous hand control device, comprising: at least one processor and a memory;
[0039] The memory stores computer-executed instructions;
[0040] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.
[0041] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods described in the first aspect and various possible designs of the first aspect.
[0042] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above and various possible designs of the first aspect.
[0043] The dexterous hand control method, device, intelligent agent, and storage medium provided in this application include: acquiring sensor data; preprocessing the sensor data to obtain input information, including environmental information and dexterous hand state information; inputting the input information into a cerebellum model to obtain corresponding control commands; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output corresponding trajectory planning results based on the input information; the motion controller is used to convert the trajectory planning results into corresponding control commands; the motion controller is obtained by combining the cerebellum model neural network with the whole-body dynamics model based on a model predictive control algorithm; and the motion controller drives and controls the actuator module of the dexterous hand according to the control commands.
[0044] The dexterous hand control method provided in this application improves the flexibility and accuracy of control by introducing a cerebellar mechanism and combining a cerebellar model neural network with a whole-body dynamics model for model predictive control. This enables the dexterous hand to perform various complex tasks more accurately, helps the dexterous hand quickly adapt to different operating environments and task requirements, reduces dependence on precise mathematical models, and thus simplifies the design and implementation of the control system. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0046] Figure 1 is a schematic diagram of a scenario for the dexterous hand control method provided in an embodiment of this disclosure;
[0047] Figure 2 is a flowchart illustrating the dexterous hand control method provided in an embodiment of this application;
[0048] Figure 3 is a flowchart illustrating the training method for the cerebellar model provided in an embodiment of this application;
[0049] Figure 4 is a schematic diagram of the structure of the trajectory planning training model provided in an embodiment of this application;
[0050] Figure 5 is a schematic diagram of the structure of the neural network to be trained in the cerebellum model provided in the embodiment of this application;
[0051] Figure 6 is a schematic diagram of the model predictive control based on the whole body dynamics model provided in an embodiment of this application;
[0052] Figure 7 is a schematic diagram of the cerebellum model provided in an embodiment of this application;
[0053] Figure 8 is a second structural schematic diagram of the cerebellum model provided in the embodiment of this application;
[0054] Figure 9 is a schematic diagram of the cerebellum model provided in the embodiment of this application.
[0055] Figure 10 is a schematic diagram of the dexterous hand control device provided in an embodiment of this application;
[0056] Figure 11 is a schematic diagram of the hardware structure of the dexterous hand control device provided in the embodiment of this application.
[0057] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0059] It should be noted that the dexterous hand control method provided in this application can be used in the field of humanoid robot technology, or in any field other than the field of humanoid robot technology. The application field of the dexterous hand control method provided in this application is not limited.
[0060] With the rapid advancement of technology, robotics is developing at an unprecedented pace, and high-degree-of-freedom dexterous hands, as outstanding representatives in this field, are remarkable for their complex and precise maneuvering capabilities. However, this high degree of freedom also brings unprecedented control challenges, especially in task scenarios requiring rapid response and high coordination. Traditional control strategies, while performing well in certain scenarios, are often limited by complex mathematical models and fixed algorithmic frameworks, making it difficult to flexibly cope with changing operating environments.
[0061] To address the aforementioned technical problems, the inventors of this application have discovered that the cerebellum, as a key part of the body responsible for motor coordination, exhibits remarkable learning and adaptive capabilities. It can adjust its movement strategies rapidly to precisely adapt to changes in the external environment. Therefore, the cerebellum's learning mechanism can be integrated into the field of robot control, particularly the control of high-degree-of-freedom dexterity hands. Introducing a cerebellum learning mechanism into the control of high-degree-of-freedom dexterity hands offers several advantages. First, it improves the flexibility and precision of control, enabling dexterity hands to perform various complex tasks more accurately. Second, due to the cerebellum's powerful adaptive capabilities, it helps dexterity hands quickly adapt to different operating environments and task requirements. Finally, this control strategy is expected to reduce reliance on precise mathematical models, thereby simplifying the design and implementation of control systems. Furthermore, to ensure accuracy, a cerebellum model neural network can be combined with a whole-body dynamics model for model predictive control (MPC), further improving control precision. Based on this, this application provides a dexterity hand control method.
[0062] Figure 1 is a schematic diagram of a scenario for the dexterous hand control method provided in this embodiment. As shown in Figure 1, the dexterous hand control system includes an intelligent agent body and a cerebellum model. The intelligent agent body may include a sensor module and an actuator module, and the cerebellum model may include a trajectory planning model and a motion controller.
[0063] In the specific implementation process, the sensor module collects sensor data, the control device acquires the sensor data, preprocesses the sensor data to obtain input information, including environmental information and dexterous hand state information; the input information is input into the cerebellum model to obtain corresponding control commands; during the cerebellum model's processing of the input information, the trajectory planning model outputs the corresponding trajectory planning result based on the input information, and the motion controller converts the trajectory planning result into the corresponding control commands, wherein the motion controller is obtained by combining the cerebellum model neural network with the whole-body dynamics model based on the model predictive control algorithm. The control device outputs the control commands (including the joint torques of the dexterous hand and the robotic arm) to the actuator module, and the actuator module drives the dexterous hand and the robotic arm according to the control commands to complete tasks such as grasping. The dexterous hand control method provided in this application, by introducing the cerebellum mechanism and combining the cerebellum model neural network with the whole-body dynamics model for model predictive control, can improve the flexibility and accuracy of control, enabling the dexterous hand to perform various complex tasks more accurately, helping the dexterous hand to quickly adapt to different operating environments and task requirements, reducing the dependence on precise mathematical models, thereby simplifying the design and implementation of the control system.
[0064] It should be noted that the scenario diagram shown in Figure 1 is merely an example. The dexterous hand control method and scenario described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0065] The technical solutions of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0066] Figure 2 is a flowchart illustrating the dexterous hand control method provided in an embodiment of this application. As shown in Figure 2, the method includes:
[0067] 201. Acquire sensor data.
[0068] The executing entity in this embodiment can be an intelligent agent such as a humanoid robot.
[0069] In this embodiment, sensor data may include environmental information (such as object position, shape, and size) and key data such as finger position, force, and speed collected by sensors such as position sensors, vision sensors, force sensors, and speed sensors.
[0070] 202. Preprocess the sensor data to obtain input information; the input information includes environmental information and dexterous hand status information.
[0071] Specifically, after obtaining sensor data, at least one of the following preprocessing methods can be performed on the sensor data: filtering, noise reduction, normalization, data cleaning, and data segmentation, to obtain input information.
[0072] For example, during filtering, different types of filters, such as low-pass, high-pass, and band-pass filters, can be selected based on the characteristics and requirements of the data to remove high-frequency noise and interference. During denoising, methods such as mean filtering, median filtering, and Kalman filtering can be used to reduce noise and outliers in the data. During normalization, methods such as linear and nonlinear normalization can be used to convert the data into values within a specific range (e.g., 0 to 1 or -1 to 1) for subsequent analysis and processing. During data cleaning, duplicate data can be removed, missing data can be processed, and erroneous data can be corrected. Duplicate data can be identified and deleted by comparing data values with timestamps; missing data can be filled using interpolation, regression, or other methods; and erroneous data can be corrected by setting thresholds or using other methods. In data segmentation, data can be segmented based on conditions such as timestamps, task type, and object type for more detailed analysis and processing.
[0073] In some embodiments, the environmental information includes at least one of the following parameters of an object in the real environment in which the dexterous hand is located: position, shape, size; the state information includes at least one of the following parameters: position, force, speed, acceleration of the dexterous hand's fingers.
[0074] 203. Input the input information into the cerebellar model to obtain the corresponding control command; the cerebellar model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output the corresponding trajectory planning result according to the input information, and the motion controller is used to convert the trajectory planning result into the corresponding control command; the motion controller is obtained by combining the cerebellar model neural network with the whole body dynamics model based on the model predictive control algorithm.
[0075] Specifically, the trained cerebellar model is input with environmental information and dexterity state information extracted from sensor data. The trajectory planning model outputs corresponding trajectory planning results based on the input information, and the motion controller converts the trajectory planning results into control commands through adaptive linear mapping.
[0076] 204. Drive and control the actuator module of the dexterous hand according to the control command.
[0077] Specifically, after receiving control commands, the system sends them to the dexterous hand's actuator module to drive the hand to perform tasks such as grasping and manipulation. During this process, motion planning and coordination are performed, including planning the dexterous hand's trajectory and coordinating finger movements to ensure the hand completes the task according to the predetermined path and actions. Real-time feedback and online adjustment mechanisms are used to continuously optimize the control strategy to adapt to changes in different environments and task requirements.
[0078] The dexterous hand control method provided in this embodiment improves the flexibility and accuracy of control by introducing a cerebellum mechanism and combining a cerebellum model neural network with a whole-body dynamics model for model predictive control. This enables the dexterous hand to perform various complex tasks more accurately, helps the dexterous hand quickly adapt to different operating environments and task requirements, reduces dependence on precise mathematical models, and thus simplifies the design and implementation of the control system.
[0079] Figure 3 is a flowchart illustrating the training method for the cerebellar model provided in this embodiment. As shown in Figure 3, the method includes:
[0080] 301. Based on the physical structure of the dexterous hand and the robotic arm, construct a physical model.
[0081] Specifically, based on the actual physical structure of the dexterous hand and robotic arm, a three-dimensional model is constructed as a physical model. This physical model can include the geometric dimensions and physical properties of fingers, joints, drive mechanisms, etc. Using kinematic and dynamic equations, kinematic and dynamic models of the dexterous hand and robotic arm are established. These models describe the motion patterns and force conditions of the dexterous hand in different postures.
[0082] For example, a kinematic model based on the Denavit-Hartenberg (DH) model can be constructed, where the forward kinematics formulas for the robotic arm and dexterous hand can be: J pose =F k (J angle (1)
[0083] Among them, J angle J is the input joint angle. pose The pose of the output robotic arm and dexterous hand.
[0084] The inverse kinematics formulas for the robotic arm and hand are: J angle =f k (J pose (2)
[0085] Among them, J angle J is the output joint angle. pose The pose of the robotic arm and dexterous hand is input.
[0086] A Lagrange-based dynamic model can be constructed: where the positive dynamic model of the robotic arm and dexterous hand is: J angle =F d (J τ (3)
[0087] Among them, J τ J is the input driving torque. angle The joint angles of the output robotic arm and dexterous hand;
[0088] The inverse dynamics model of robotic arms and dexterous hands is: J τ =f d (J angle (4)
[0089] Among them, J τ J is the output driving torque. angle The joint angles of the robotic arm and dexterous hand are input.
[0090] 302. Construct the cerebellum neural network to be trained and the trajectory planning model to be trained.
[0091] Specifically, the trajectory planning model to be trained employs imitation learning methods to extract trajectory features from human or other dexterous hand manipulation data. Based on deep learning techniques (such as Convolutional Neural Networks (CNNs) and attention-based sequence models like Transformers), the trajectories are encoded and decoded to generate trajectories adapted to different tasks. CNNs can be used to extract spatial features of the trajectory, while Transformers can capture the temporal dependencies of the trajectory.
[0092] In some embodiments, the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7. This trajectory planning system combines the advantages of CNN and Transformer, and can effectively extract information from image sequences and predict future robotic arm and dexterous hand trajectories. The system has good scalability and flexibility, and can be adjusted and optimized according to specific application scenarios.
[0093] Specifically, the input to the trajectory planning model to be trained includes image sequences (visual information acquired through visual sensors) and the corresponding positions of the robotic arm and dexterous hand (state information acquired through position sensors, etc.), and random variables. The output of the trajectory planning model to be trained includes the predicted trajectory sequence and random variables.
[0094] For example, as shown in Figure 4, the input to the trajectory planning system can be defined as the image sequence X = {x1, x2, ..., x...} n}, x n Let x be the nth image, where n is a positive integer greater than 1, and x be the information of the i-th image. i ∈R w×h×d w, h, d represent the width, height, and depth of the image (e.g., the size could be 3, 480, 630 respectively). The output of the trajectory planning system can be defined as trajectory sequence information T = {t1, t2, ..., t...} n},t n Let n be the nth trajectory, where n is a positive integer greater than 1. Each trajectory includes the pose of the robotic arm's end effector (e.g., 7-dimensional) and the bending angle of the dexterous hand (e.g., 15-dimensional). The trajectory planning system can employ the Mobile A Low-cost Open-source Hardware System (Mobile ALHOA) algorithm, with a backbone network consisting of a CNN and a Transformer. The CNN uses ResNet18, the Transformer encoder uses a 4-layer self-attention network, and the Transformer decoder uses a 7-layer cross-attention network.
[0095] The cerebellar neural network to be trained can include an input layer, intermediate layers (such as a cerebellar cortical simulation layer), and an output layer. The intermediate layers can contain multiple cerebellar units, each of which performs nonlinear processing on the input through sparse connections and local generalization capabilities.
[0096] For example, as shown in Figure 5, the input to the cerebellar neural network to be trained is trajectory information, and the output is information such as joint torques. The cerebellar neural network to be trained includes a virtual associative computing (AC) space and a physical storage (AP) space. Through the conceptual and practical mapping of these two spaces, nonlinear mapping is achieved based on local approximation and associative functions. The virtual associative computing (AC) space is a mapping of the input space, used to store the quantized encoded information of the input state. After quantization, the input vector is mapped to multiple storage units in the AC. These storage units partially overlap; similar inputs will activate more overlapping units in the AC. The physical storage (AP) space stores the weights used for network training. Each storage unit in the AC corresponds to a weight in the AP. The output of the network is the sum of the weights of all activated storage units in the AP. Compared to traditional control algorithms, the control algorithm based on the cerebellar model has faster efficiency and accuracy.
[0097] 303. Obtain the training sample set.
[0098] Specifically, a teleoperation system can be used to control dexterous hands and robotic arms to perform grasping and manipulation tasks in different scenarios (such as objects of different shapes, materials, and postures, and different operating environments). Real-time acquisition of environmental information (such as object position, shape, and size), finger position, force, and speed sensor data is crucial. The acquired data is then preprocessed (e.g., filtering, noise reduction, normalization) to improve data quality and usability.
[0099] In some embodiments, to ensure the authenticity and richness of the training data and improve training accuracy, data collected specifically from different scenarios can be used as training samples. Specifically, obtaining the training sample set may include: controlling a dexterous hand and a robotic arm to perform grasping and manipulation tasks in multiple scenarios through a teleoperation system, and collecting sensor data during task execution; the task objects or operating environments are different in different scenarios; different task objects refer to task objects having different parameters of at least one of the following: shape, material, and position; preprocessing the sensor data to obtain the training sample set.
[0100] 304. Based on the training sample set and the physical model, train the cerebellar neural network to be trained and the trajectory planning model to be trained to obtain the cerebellar model neural network and the trajectory planning model.
[0101] Specifically, during training, sensor data acquired through teleoperation can be input into the trajectory planning model to be trained. The model's CNN and transformer are then used to generate predicted trajectory features. These predicted trajectory features, combined with environmental and dexterity hand state information, are then input into the motion controller model. The motion controller model (cerebellar neural network to be trained, whole-body dynamics model, and model predictive control) outputs corresponding control commands based on the input information. These commands are converted into signals understandable to the dexterity hand actuator through adaptive linear mapping. After obtaining the corresponding control commands, the error between the actual results obtained from the physical model (such as the dexterity hand's trajectory and force) and the expected results is used as a loss function, and the model's weights are updated through backpropagation. Based on the algorithm's convergence and performance, the model is optimized and adjusted, including adjusting the learning rate, increasing the number of iterations, and modifying the network structure (such as increasing or decreasing the number of cerebellar units or changing the connection pattern).
[0102] In some embodiments, to improve training efficiency, the cerebellar neural network to be trained and the trajectory planning model to be trained can be trained separately. Specifically, the training sample set includes multiple sets of image sequences and corresponding expected trajectories, as well as multiple sets of trajectories and corresponding expected torques; training the cerebellar neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model may include: training the cerebellar neural network to be trained based on a first loss function and a backpropagation algorithm, according to multiple sets of trajectories and corresponding expected torques, to obtain the cerebellar model neural network; the first loss function includes the error between the actual torque output by the cerebellar neural network to be trained and the expected torque; training the trajectory planning model to be trained based on a second loss function and a backpropagation algorithm, according to multiple sets of image sequences and corresponding expected trajectories, to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.
[0103] For example, training can be performed using simulated multi-joint dynamics with contact (MuJoCo). Model training can utilize a 12GB RTX 3080 graphics processing unit (GPU). A Smooth L1 loss function (Huber loss function) and an Adaptive Moment Estimation (Adam) W optimizer are employed. The expression for the loss function is as follows:
[0104] Where Loss is the loss function, T pred T represents the model's predicted value, such as the desired torque or desired trajectory. true This represents the actual value, such as the actual torque or the actual trajectory.
[0105] During training, the trajectory planning model is trained first for 100 epochs, with each batch consisting of 32 image sequences and their corresponding trajectories. Then, the cerebellum neural network is trained, with each batch consisting of 128 trajectories and their corresponding moments. In each training iteration, state-action-reward sequences are generated from the MuJoCo environment. These sequences are used to update the corresponding models. The loss is calculated based on the corresponding loss function, and the AdamW optimizer is used to update the model weights. Model performance is periodically checked, and hyperparameters such as the learning rate and weight decay are adjusted as needed.
[0106] In some embodiments, to improve training accuracy while ensuring training efficiency, the cerebellar model can be initially trained, and then the initial cerebellar model neural network and the trajectory planning model to be trained obtained from the initial training can be jointly trained. Specifically, the training sample set includes multiple sets of trajectories and corresponding expected moments, as well as multiple sets of image sequences and corresponding expected moments; the step of training the cerebellar model neural network and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model can include: training the cerebellar model neural network to be trained based on a first loss function and a backpropagation algorithm, according to multiple sets of trajectories and corresponding expected moments, to obtain an initial cerebellar model neural network; the first loss function includes the error between the actual moment and the expected moment output by the cerebellar model neural network to be trained; and jointly training the trajectory planning model and the initial cerebellar model neural network based on a second loss function and a backpropagation algorithm, according to multiple sets of image sequences and corresponding expected moments, to obtain the trajectory planning model and the cerebellar model neural network; the second loss function includes the error between the actual trajectory and the expected trajectory output by the trajectory planning model to be trained.
[0107] For example, training can be performed using the simulation environment MuJoCo. Model training can be performed using a 12G RTX 3080 GPU. The Smooth L1 loss function (Huber loss function, as shown in expression (5)) and the AdamW optimizer are used.
[0108] During training, the cerebellum training neural network is trained first, with 128 trajectories and corresponding torques input in each batch. In each training iteration, state-action-reward sequences are generated from the MuJoCo environment. The loss is calculated based on the corresponding loss function, and the Adam W optimizer is used to update the model's weights. The model's performance is checked periodically, and hyperparameters such as the learning rate and weight decay are adjusted as needed. Then, the training trajectory planning model and the cerebellum training neural network are trained together for 100 epochs, with 32 sets of image sequences and corresponding torques input in each batch. In each training iteration, state-action-reward sequences are generated from the MuJoCo environment. The loss is calculated based on the corresponding loss function, and the Adam W optimizer is used to update the model's weights. The model's performance is checked periodically, and hyperparameters such as the learning rate and weight decay are adjusted as needed.
[0109] 305. Based on the model predictive control algorithm, the cerebellar model neural network is combined with the whole-body dynamics model to obtain a motion controller, and the cerebellar model is determined according to the motion controller and the trajectory planning model.
[0110] Specifically, to further improve accuracy, a motion controller can be constructed by combining a cerebellar model neural network with model predictive control (MPC) and a whole-body dynamics model. MPC methods are used to predict future states and optimize control strategies, while the whole-body dynamics model provides an accurate description of dexterous hand movements. Optimal control strategies are then solved using optimization algorithms (such as gradient descent or genetic algorithms) to achieve precise motion control. Simultaneously, the rapid learning and adaptive capabilities of the cerebellar model are utilized to adjust and optimize the control strategy online, adapting to changes in different environments and task requirements.
[0111] In some embodiments, there are multiple ways to combine a cerebellar model neural network with a whole-body dynamics model based on a model predictive control algorithm. In one possible implementation, to improve computational accuracy, the two models can be combined in parallel. Specifically, the combination of the cerebellar model neural network and the whole-body dynamics model based on the model predictive control algorithm to obtain a motion controller may include: connecting an inverse kinematics module and an inverse dynamics module in series and then connecting them in parallel with the cerebellar model neural network to obtain a motion controller; wherein, the inverse dynamics module is also used to connect in series with the forward dynamics module and the forward kinematics module in sequence, the forward dynamics module is used to provide feedback output to the inverse dynamics module, and the forward kinematics module is used to provide feedback output to the inverse kinematics module and the cerebellar model neural network. In this embodiment, by combining the cerebellar model neural network with the whole-body dynamics model in parallel, the information about the actual motion state (such as joint angles, end effector positions, etc.) output by the forward kinematics module is fed back to the cerebellar model neural network. This allows the cerebellar model neural network to adjust its internal weights and parameters to correct the motion trajectory and adapt to the dynamic characteristics, thereby more accurately predicting and generating control signals, improving control precision, enhancing robustness, and optimizing motion planning. Furthermore, the parallel combination method can combine the cerebellar model and the inverse kinematics output to improve accuracy.
[0112] In another possible implementation, the two models can be concatenated to improve computational efficiency. Specifically, the model predictive control algorithm, which combines a cerebellar model neural network with a whole-body dynamics model to obtain a motion controller, can include: sequentially concatenating the cerebellar model neural network with a positive dynamics module and a positive kinematics module to obtain the motion controller; wherein the positive kinematics module is used to provide feedback output to the cerebellar model neural network. In this embodiment, by concatenating the cerebellar model neural network with the whole-body dynamics model, the information about the actual motion state (e.g., joint angles, end effector position, etc.) output by the positive kinematics module is fed back to the cerebellar model neural network. This allows the cerebellar model neural network to adjust its internal weights and parameters to correct the motion trajectory and adapt to dynamic characteristics, thereby more accurately predicting and generating control signals, improving control accuracy, enhancing robustness, optimizing motion planning, and simplifying the model structure and improving computational efficiency by using a concatenated approach.
[0113] For example, as shown in Figure 6, the model predictive control algorithm based on whole-body dynamics solves for control commands through inverse kinematics and dynamics, and adjusts the control commands to maintain consistency with the desired trajectory through forward kinematics and dynamics feedback.
[0114] For example, as shown in Figures 7 and 8, the accuracy of control can be improved by combining a cerebellar model with MPC based on whole-body dynamics.
[0115] As shown in Figure 7, the cerebellar model neural network can be connected in parallel with the whole-body dynamics-based MPC. Specifically, while the cerebellar model neural network generates control commands based on the predicted trajectory output by the trajectory planning model, control commands can also be generated through inverse kinematics and inverse dynamics modules. These two control commands are then fused to obtain the final control command. Simultaneously, feedback adjustment based on forward dynamics and forward kinematics can further improve the accuracy of the control commands.
[0116] As shown in Figure 8, the cerebellar model neural network can be connected in series with the whole-body dynamics-based MPC. Specifically, after the cerebellar model neural network generates control commands based on the predicted trajectory output by the trajectory planning model, the accuracy of the control commands can be further improved through feedback adjustment of positive dynamics and positive kinematics.
[0117] In some embodiments, as shown in Figure 9, in order to simplify the model and improve computational efficiency, the cerebellar model neural network can also be used as a motion controller. Then, a cerebellar model is constructed based on the trajectory planning model and the cerebellar model neural network. The cerebellar model neural network processes the predicted trajectory output by the trajectory planning model to obtain control commands.
[0118] The cerebellar model training method provided in this embodiment combines the cerebellar model neural network with MPC based on the whole-body dynamics model to obtain a motion controller. Based on the motion controller and the trajectory planning model, a cerebellar model is obtained, thereby introducing the cerebellar mechanism into dexterous hand control. This can improve the flexibility and accuracy of control, enabling dexterous hands to perform various complex tasks more accurately. It also helps dexterous hands quickly adapt to different operating environments and task requirements, reduces the dependence on precise mathematical models, and simplifies the design and implementation of the control system.
[0119] Figure 10 is a schematic diagram of the structure of the dexterous hand control device provided in an embodiment of this application. As shown in Figure 10, the dexterous hand control device 100 includes: an acquisition module 1001, a determination module 1002, a processing module 1003, and a drive control module 1004.
[0120] Acquisition module 1001 is used to acquire sensor data;
[0121] The determination module 1002 is used to preprocess the sensor data to obtain input information; the input information includes environmental information and dexterous hand status information.
[0122] Processing module 1003 is used to input the input information into the cerebellum model to obtain corresponding control commands; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output corresponding trajectory planning results according to the input information, and the motion controller is used to convert the trajectory planning results into corresponding control commands; the motion controller is obtained by combining the cerebellum model neural network with the whole-body dynamics model based on the model predictive control algorithm;
[0123] The drive control module 1004 is used to drive and control the actuator module of the dexterous hand according to the control command.
[0124] The dexterous hand control device provided in this application, by simulating the efficient learning and adaptive capabilities of the cerebellum, can perceive changes in the external environment and the complexity of the operational task in real time, and dynamically adjust and optimize the control strategy accordingly. This capability significantly improves the adaptability and flexibility of the dexterous hand in complex and ever-changing environments, enabling it to flexibly cope with various challenges. By introducing the cerebellum learning framework, the system can more accurately match the operational environment and task requirements. This not only improves the operational accuracy of the dexterous hand but also significantly enhances its grasping stability, ensuring the smooth completion of the operational task. Furthermore, the rapid response capability of the cerebellum learning mechanism allows the system to quickly process data and make decisions after receiving sensor data, generating control commands. This rapid response capability is particularly important for tasks that require urgent adjustments or responses to unexpected situations, and can significantly improve overall operational efficiency.
[0125] In some embodiments, the environmental information includes at least one of the following parameters of an object in the real environment in which the dexterous hand is located: position, shape, size; the state information includes at least one of the following parameters: position, force, speed, acceleration of the dexterous hand's fingers.
[0126] In some embodiments, the device 100 further includes a training module (not shown): constructing a physical model based on the physical structure of the dexterous hand and the physical structure of the robotic arm; constructing a cerebellar neural network to be trained and a trajectory planning model to be trained; acquiring a training sample set; training the cerebellar neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model; combining the cerebellar model neural network with a whole-body dynamics model based on a model predictive control algorithm to obtain a motion controller; and determining the cerebellar model based on the motion controller and the trajectory planning model.
[0127] In some embodiments, the training module is specifically used to: connect the inverse kinematics module and the inverse dynamics module in series and then connect them in parallel with the cerebellar model neural network to obtain a motion controller; wherein, the inverse dynamics module is also used to connect in series with the forward dynamics module and the forward kinematics module in sequence, the forward dynamics module is used to provide feedback output to the inverse dynamics module, and the forward kinematics module is used to provide feedback output to the inverse kinematics module and the cerebellar model neural network.
[0128] In some embodiments, the training module is specifically used to: connect the cerebellar model neural network in series with the positive dynamics module and the positive kinematics module to obtain a motion controller; wherein the positive kinematics module is used to provide feedback output to the cerebellar model neural network.
[0129] In some embodiments, the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7.
[0130] In some embodiments, the training module is specifically used to: control a dexterous hand and a robotic arm to perform grasping and manipulation tasks in multiple scenarios through a teleoperation system, and collect sensor data during task execution; the task objects or operating environments are different in different scenarios; different task objects refer to task objects having different parameters of at least one of the following: shape, material, and position; preprocess the sensor data to obtain a training sample set.
[0131] In some embodiments, the training sample set includes multiple sets of image sequences and corresponding expected trajectories, as well as multiple sets of trajectories and corresponding expected moments; the training module is specifically used to: train the cerebellar neural network to be trained based on a first loss function and a backpropagation algorithm, according to the multiple sets of trajectories and corresponding expected moments, to obtain the cerebellar model neural network; the first loss function includes the error between the actual moment output by the cerebellar neural network to be trained and the expected moment; and train the trajectory planning model to be trained based on a second loss function and a backpropagation algorithm, according to the multiple sets of image sequences and corresponding expected trajectories, to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.
[0132] In some embodiments, the training module is specifically used for: the training sample set including multiple sets of trajectories and corresponding expected moments, and multiple sets of image sequences and corresponding expected moments; the training module is specifically used for: training the cerebellar neural network to be trained based on a first loss function and a backpropagation algorithm, according to the multiple sets of trajectories and corresponding expected moments, to obtain an initial cerebellar model neural network; the first loss function includes the error between the actual moment output by the cerebellar neural network to be trained and the expected moment; and jointly training the trajectory planning model to be trained and the initial cerebellar model neural network based on a second loss function and a backpropagation algorithm, according to the multiple sets of image sequences and corresponding expected moments, to obtain the trajectory planning model and the cerebellar model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.
[0133] The dexterous hand control device provided in this application embodiment can be used to execute the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.
[0134] Figure 11 is a schematic diagram of the hardware structure of the dexterous hand control device provided in the embodiment of this application.
[0135] Device 110 may include one or more of the following components: processing component 1101, memory 1102, power supply component 1103, multimedia component 1104, audio component 1105, input / output (I / O) interface 1106, sensor component 1107, and communication component 1108.
[0136] Processing component 1101 typically controls the overall operation of device 110, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1101 may include one or more processors 1109 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1101 may include one or more modules to facilitate interaction between processing component 1101 and other components. For example, processing component 1101 may include a multimedia module to facilitate interaction between multimedia component 1104 and processing component 1101.
[0137] Memory 1102 is configured to store various types of data to support the operation of device 110. Examples of this data include instructions for any application or method operating on device 110, contact data, phonebook data, messages, pictures, videos, etc. Memory 1102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0138] Power supply component 1103 provides power to various components of device 110. Power supply component 1103 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 110.
[0139] Multimedia component 1104 includes a screen that provides an output interface between the device 110 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1104 includes a front-facing camera and / or a rear-facing camera. When the device 110 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0140] Audio component 1105 is configured to output and / or input audio signals. For example, audio component 1105 includes a microphone (MIC) configured to receive external audio signals when device 110 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1102 or transmitted via communication component 1108. In some embodiments, audio component 1105 also includes a speaker for outputting audio signals.
[0141] I / O interface 1106 provides an interface between processing component 1101 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0142] Sensor assembly 1107 includes one or more sensors for providing status assessments of various aspects of device 110. For example, sensor assembly 1107 may detect the on / off state of device 110, the relative positioning of components such as the display and keypad of device 110, changes in the position of device 110 or a component of device 110, the presence or absence of user contact with device 110, the orientation or acceleration / deceleration of device 110, and temperature changes of device 110. Sensor assembly 1107 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1107 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1107 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0143] Communication component 1108 is configured to facilitate wired or wireless communication between device 110 and other devices. Device 110 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1108 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1108 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0144] In an exemplary embodiment, device 110 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0145] This application also provides an intelligent agent, including a dexterous hand, an arm, sensors, an actuator, and a dexterous hand control device as described in the above embodiments.
[0146] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1102 including instructions, which can be executed by a processor 1109 of device 110 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0147] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0148] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0149] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0150] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the dexterous hand control method executed by the dexterous hand control device described above.
[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for controlling a dexterous hand, characterized in that, include: Acquire sensor data; The sensor data is preprocessed to obtain input information; The input information includes environmental information and the state information of the dexterous hand; The input information is input into the cerebellum model to obtain the corresponding control commands; The cerebellum model includes a trajectory planning model and a motion controller; The trajectory planning model is used to output the corresponding trajectory planning result based on the input information, and the motion controller is used to convert the trajectory planning result into the corresponding control command; the motion controller is obtained by combining the cerebellar model neural network with the whole-body dynamics model based on the model predictive control algorithm; The actuator module of the dexterous hand is driven and controlled according to the control command.
2. The method according to claim 1, characterized in that, The environmental information includes at least one of the following parameters of objects in the real environment where the dexterous hand is located: position, shape, and size; the state information includes at least one of the following parameters: position, force, speed, and acceleration of the dexterous hand's fingers; before inputting the input information into the cerebellum model to obtain the corresponding control command, the process further includes: Based on the physical structure of the dexterous hand and the robotic arm, construct a physical model; Construct a cerebellar neural network to be trained and a trajectory planning model to be trained; the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7. Obtain the training sample set; Based on the training sample set and the physical model, the cerebellar neural network to be trained and the trajectory planning model to be trained are trained to obtain the cerebellar model neural network and the trajectory planning model; The cerebellar model neural network is combined with the whole-body dynamics model based on the model predictive control algorithm to obtain a motion controller, and the cerebellar model is determined according to the motion controller and the trajectory planning model.
3. The method according to claim 2, characterized in that, The model predictive control algorithm combines a cerebellar model neural network with a whole-body dynamics model to obtain a motion controller, including: The inverse kinematics module and the inverse dynamics module are connected in series and then connected in parallel with the cerebellar model neural network to obtain a motion controller. The inverse dynamics module is also used to connect in series with the forward dynamics module and the forward kinematics module in sequence. The forward dynamics module is used to provide feedback output to the inverse dynamics module, and the forward kinematics module is used to provide feedback output to the inverse kinematics module and the cerebellar model neural network.
4. The method according to claim 2, characterized in that, The model predictive control algorithm combines a cerebellar model neural network with a whole-body dynamics model to obtain a motion controller, including: A motion controller is obtained by sequentially connecting a cerebellar model neural network with a positive dynamics module and a positive kinematics module; wherein, the positive kinematics module is used to provide feedback output to the cerebellar model neural network.
5. The method according to any one of claims 2-4, characterized in that, The acquisition of the training sample set includes: The system remotely controls a dexterous hand and robotic arm to perform grasping and manipulation tasks in multiple scenarios, and collects sensor data during task execution. The task objects or operating environments are different in different scenarios. Different task objects refer to task objects that have different parameters of at least one of the following: shape, material, or position. The sensor data is preprocessed to obtain a training sample set.
6. The method according to any one of claims 2-5, characterized in that, The training sample set includes multiple sets of image sequences and corresponding expected trajectories, as well as multiple sets of trajectories and corresponding expected torques; the step of training the cerebellar neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model includes: Based on the first loss function and the backpropagation algorithm, the cerebellar neural network to be trained is trained according to multiple sets of trajectories and corresponding expected torques to obtain the cerebellar model neural network; the first loss function includes the error between the actual torque and the expected torque output by the cerebellar neural network to be trained. Based on the second loss function and the backpropagation algorithm, the trajectory planning model is trained according to multiple sets of image sequences and corresponding expected trajectories to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model and the expected trajectory.
7. The method according to any one of claims 2-6, characterized in that, The training sample set includes multiple sets of trajectories and corresponding expected moments, as well as multiple sets of image sequences and corresponding expected moments; the step of training the cerebellar neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellar model neural network and the trajectory planning model includes: Based on the first loss function and the backpropagation algorithm, the cerebellar neural network to be trained is trained according to multiple sets of trajectories and corresponding expected torques to obtain an initial cerebellar model neural network; the first loss function includes the error between the actual torque and the expected torque output by the cerebellar neural network to be trained. Based on the second loss function and the backpropagation algorithm, the trajectory planning model to be trained and the initial cerebellar model neural network are jointly trained according to multiple sets of image sequences and corresponding expected moments to obtain the trajectory planning model and the cerebellar model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.
8. A dexterous hand control device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the dexterous hand control method as described in any one of claims 1 to 7.
9. An intelligent agent, characterized in that, It includes a dexterous hand, an arm, sensors, actuators, and a dexterous hand control device as described in claim 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the dexterous hand control method as described in any one of claims 1 to 7.