200 results about "Hand joint" patented technology

The invention discloses a skeleton sign language recognition method of a double-flow space-time dynamic graph convolutional network fused with residual learning, and belongs to the technical field of artificial intelligence and gesture recognition. According to the method, an input gesture skeleton sequence relative to a face is divided into two data streams, namely a hand form data stream and a wrist track data stream through double-reference-system differential homeomorphic mapping; the method comprises the following steps: firstly, processing hand posture data, and capturing a hand joint spatial topological relation by combining a spatial-temporal dynamic graph convolutional network (STDGCNN) with a residual convolutional block; meanwhile, a Finsler trajectory dynamics encoder (FTDE) is adopted to carry out differential geometric modeling on the wrist trajectory, and the direction sensitivity characteristic of the trajectory is captured through a multi-scale causal convolutional network. Then, mutual enhancement of double-flow features is realized through a bidirectional cross feature enhancer (BCFE), and the problem of geometric inconsistency of a heterogeneous feature space is solved through a geometric-driven optimal transmission fusion device (Geometric-OT). The method solves the challenge that a traditional sign language recognition method processes complex space-time correlation of gesture forms and motion tracks at the same time, the technical problems of insufficient relation between hands and faces, insufficient feature expression ability and low space-time feature extraction efficiency, and the problems of geometric inconsistency, single reference system and the like. And the identification accuracy and the real-time performance are obviously improved. Experiments show that the accuracy rate of the method in complex hundreds of sign language vocabulary recognition tasks reaches 95% or above on average, the reasoning speed is only 17ms on average, high-precision real-time sign language recognition is achieved, and the method has higher robustness in complex environments such as noise and shielding.

An augmented reality device for obtaining three-dimensional (3D) position information of a plurality of hand joints of a user includes a plurality of cameras configured to photograph the user's hand and obtain images; memory storing instructions; and at least one processor; the instructions, when executed, may cause the device to recognize the plurality of hand joints from the images; obtain two-dimensional (2D) joint coordinate values for feature points corresponding to the hand joints; retrieve, from a look up table (LUT), 3D position coordinate values corresponding to distortion model parameters of the cameras, a positional relationship between the cameras, and the 2D joint coordinate values; and output the 3D position information of the plurality of hand joints based on the 3D position coordinate values.

The invention relates to a biomechanical modeling and normalized training data generation method and device for human hand posture estimation, equipment and a medium. The method comprises the following steps: acquiring target three-dimensional hand posture data of a target object; for each target object, acquiring reference static biomechanical characteristics and dynamic biomechanical characteristics corresponding to each target three-dimensional hand posture of the target object; for each target three-dimensional hand posture, determining a three-dimensional joint coordinate of the target three-dimensional hand posture and a two-dimensional joint coordinate corresponding to the three-dimensional joint coordinate based on the reference static biomechanical characteristics and the dynamic biomechanical characteristics; wherein the three-dimensional joint coordinates represent the spatial positions of the hand joint points; and determining hand posture training data based on the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates. By adopting the method, the individual biomechanical structure difference can be eliminated, and the number of training data can be expanded more accurately and reasonably.

A hand pose recognition method is performed by a computer device, including: acquiring a current frame of a multi-lens video of a target object; performing hand detection on a first view of the current frame to obtain a first lens detection result; performing hand estimation on a second view of the current frame to obtain a second lens estimation result; removing, from the hand detection boxes in the first view and the hand estimation boxes in the second view, redundant boxes corresponding to redundant hands, and then performing hand joint point recognition on remaining boxes to obtain two-dimensional joint points; converting the two-dimensional joint points into three-dimensional joint points in a three-dimensional hand coordinate system; and converting the three-dimensional joint points in the three-dimensional hand coordinate system into three-dimensional joint points of the current frame in a world coordinate system according to pose estimation parameters corresponding to the current frame.

The invention discloses a self-adaptive exoskeleton robot which comprises a hand exoskeleton device used for capturing the degree of freedom of hand joints of 20 persons at the same time and conducting force feedback. The perception and force feedback system is used for data acquisition and master control driving and is matched with IO resources of a master control chip; the self-adaptive wearing module is used for sewing the finger exoskeleton to a light, thin and elastic fiber glove, the wearing mode is reinforced through a restraint ring, and meanwhile, an exoskeleton palm structure is designed to be a variable structure capable of being attached to the back of the hand in a self-adaptive mode; the man-machine interaction module is used for man-machine interaction; and the master-slave control system realizes master-slave control. According to the invention, the bionic flexible muscle optimization design, the force feedback technology and the touch force sensor integrated with force feedback are adopted, so that the force of a user can be acquired, and a foundation is laid for force closed-loop control.

The invention discloses a stroke hand rehabilitation training method based on multi-modal data fusion, particularly relates to the field of hand rehabilitation training, and is used for solving the problems that in the existing rehabilitation process, the motion solidification risk cannot be recognized, and the training scheme is lack of individualized dynamic adjustment. According to the method, hand joint motion features are obtained by constructing a standardized hand function micro-evaluation task, and a personal motion mode baseline is established in combination with dynamic time warping and multi-scale permutation entropy analysis; extracting a neural motion curing index based on the difference change between the current motion mode and the optimal motion mode, and analyzing and generating a function recovery momentum feature by combining the rehabilitation progress gradient and the acceleration inflection point; and fusing the electromyographic signal coupling coordination index to generate a rehabilitation risk score, and when the risk exceeds a dynamic threshold value, triggering adaptive recombination of a training strategy to realize accurate early warning and personalized training regulation and control. Rehabilitation monitoring precision and training pertinence can be effectively improved, and high-quality recovery of stroke hand functions is promoted.

A control device controls a substrate transfer robot including a hand, a joint, and a joint motor. The control device modifies a target position of the hand when it places substrate at transfer destination, based on positional misalignment information indicating misalignment of substrate. To change a position at which the hand places the substrate to target position after modification, the control device performs first velocity control and second velocity control. In first velocity control, velocity of hand that is approaching target position before modification with the velocity decreasing is made constant by correcting. In second velocity control, velocity of hand decreases by slope equal to the slope of velocity of the hand immediately before first velocity control and hand is stopped at target position after modification. Time during which the first velocity control is performed increases as a magnitude of change in the target position due to the modification increases.

The invention provides a multi-source heterogeneous data synchronous mechanical arm bionic control method and device and electronic equipment, and relates to the technical field of robot control, and the method comprises the following steps: decoding hand joint angle data and a synchronization timestamp thereof, wrist pose data and a synchronization timestamp thereof, and arm pose data and a synchronization timestamp thereof, and obtaining a decoding result; the decoded hand joint angle data, the decoded wrist pose data and the decoded arm pose data are transmitted to a mechanical arm controller through a low-delay network; and the mechanical arm controller executes dynamic remapping based on the decoded hand joint angle data, the decoded wrist pose data and the decoded arm pose data, and simulation control of the mechanical arm is achieved. According to the method, the arm pose data participates in dynamic correction of the base coordinate system, space positioning deviation is eliminated, and the mechanical arm reproduces actions; timing sequence block optimization considers fine grabbing and long track actions, and action restoration authenticity is improved.

The invention discloses a smart type library-based teleoperation method, system and device, and a storage medium. The method comprises the following steps of: constructing a smart operation type library; analyzing the task instruction through a preset multi-modal large language model MLLM to obtain a corresponding operation step, and matching a dexterous operation type corresponding to the current step from a dexterous operation type library; intuitive accurate control between the hand and the manipulator is realized through an interpolation mapping strategy. The interpolation mapping strategy specifically comprises the following steps: capturing a fingertip position of a human hand in real time; calculating the normalized projection proportion of the current posture of the human hand relative to the stretching state and the contraction state; and according to the projection proportion, linear interpolation is carried out on the extension and contraction joint angles of the manipulator, and a manipulator joint angle instruction is generated. According to the method, the adaptability and stability of teleoperation are provided by establishing a systematized smart type library and combining a type automatic retrieval module assisted by a multi-modal large language model and an interpolation mapping control strategy.

The invention provides a gesture interaction method and system, a readable storage medium and a computer. The method comprises the following steps: acquiring a sight focusing area of a user for a virtual marker; inputting the hand image into a convolutional neural network model for convolution operation to obtain a convolutional feature image; performing pixel processing based on the hand coding image of the hand image to obtain a corresponding pixel feature image; importing the multi-scale fusion unit into a multi-resolution network model for model optimization to obtain a network optimization model; and inputting the convolutional feature image and the pixel feature image into a network optimization model to output a hand posture estimation result of the user, and performing collision judgment according to the hand posture estimation result and the sight focusing area to realize gesture interaction of the user. According to the method, attitude estimation is carried out on the convolution feature image and the pixel feature image by using the network optimization model, and the influence of network parameter quantity on hand attitude estimation is reduced, so that the detection error of hand joint points is reduced.

The invention relates to the technical field of intelligent robots, in particular to a virtual-real combined mechanical arm-dexterous hand system mapping operation and sample collection method. Comprises: configuring a mapping operation system; collecting the tail end position and posture information of the tail end of the forearm of the operator in real time based on the teleoperation device; acquiring three-dimensional information of key points of hands of an operator in real time based on a camera; a mechanical arm tail end mapping value is calculated according to the tail end position and the posture information; calculating a joint mapping value of the dexterous hand according to the three-dimensional information of the key points; and the mechanical arm and the dexterous hand are driven to move in real time according to the mechanical arm tail end mapping value and the dexterous hand joint mapping value, real-time data of all joints in the mapping operation process and observation images in the operation process are recorded, and sample collection is completed. The embodiment of the invention provides a virtual and real combined mechanical arm-dexterous hand system mapping operation and sample collection method, which can be used for carrying out mechanical arm-dexterous hand system mapping operation and sample collection.

The invention relates to the technical field of visual inspection, and particularly provides a method for detecting a hand joint in glasses equipment, a storage medium, electronic equipment and a product, and the method can comprise the steps: obtaining the confidence coefficient of a hand region corresponding to a current frame image; under the condition that the confidence coefficient is determined to be greater than a preset threshold value, processing the hand region to obtain a to-be-detected hand region image; and inputting the to-be-detected hand region image into a target hand joint detection model to obtain a hand joint detection result. According to some embodiments of the invention, the accuracy of hand joint detection can be improved.

The embodiment of the invention provides a robot control method and system based on virtual reality and hand motion capture, and the method comprises the steps: collecting the hand posture data of a user, obtaining the first hand posture data of the user through a virtual reality device, the first hand posture data comprises the hand end posture information, and obtaining the first hand posture data of the user through a virtual reality device; obtaining second hand posture data of the user through wearable hand motion capture equipment, wherein the second hand posture data comprises hand joint fine motion information; standardizing the hand position and posture data, and mapping the hand position and posture data into robot target motion data based on a preset calibration matrix, the robot target motion data including first robot target motion data mapped based on the first hand position and posture data and second robot target motion data mapped based on the second hand position and posture data; robot motion control data is generated based on the robot target motion data, and robot parts are controlled to move. According to the robot control method and system based on virtual reality and hand motion capture, the operation precision of the robot can be greatly improved, the operation real-time performance and smoothness are enhanced, and the motion state of the robot can be monitored in real time.

The invention discloses a hand posture reconstruction method, system and device based on sEMG and a medium. The method comprises the following steps that a forearm sEMG signal to be estimated is collected; performing denoising processing on the forearm sEMG signal based on a continuous denoising network, and segmenting the forearm sEMG signal after denoising processing into a plurality of continuous signals; performing feature extraction on the continuous signals based on a mixed attention mechanism to obtain continuous signal features; hand posture prediction based on posture correction processing is carried out on the continuous signal features based on a dual-decoder prediction architecture, and hand continuous posture signals are obtained; calculating a hand joint angular acceleration based on the hand continuous attitude signal, and converting a hand attitude reconstruction image according to the hand joint angular acceleration; according to the method, the sEMG features are extracted more effectively by fusing sparse attention and channel attention mechanisms, the attitude correction capability is remarkably enhanced by using a double-decoder structure, and the real-time accuracy of hand attitude estimation is improved.

The invention relates to the technical field of data processing, in particular to a rehabilitation training control method and system based on a flexible hand joint rehabilitation glove, and the method comprises the steps: obtaining an individualized physiological parameter baseline of a patient; real-time data of a patient in the rehabilitation training process are collected through a multi-modal sensor array on the flexible glove, the active participation degree, the joint movement smoothness and the muscle fatigue degree of the patient are calculated based on the real-time data, the active movement intention of the patient is recognized, and a real-time rehabilitation state evaluation result of the patient is obtained; generating a control instruction through an adaptive control engine according to a preset personalized rehabilitation scheme and the real-time rehabilitation state evaluation result; the generated control instruction is sent to a driving control circuit so as to control a driving unit to execute corresponding bending or stretching actions and assist the patient in completing target training actions; according to the invention, personalized, self-adaptive and quantitative evaluation of rehabilitation training can be realized, and the efficiency and effect of hand rehabilitation training are improved.

A hand pose recognition method is performed by a computer device, including: acquiring a current frame of a multi-lens video of a target object; performing hand detection on a first view of the current frame to obtain a first lens detection result; performing hand estimation on a second view of the current frame to obtain a second lens estimation result; removing, from the hand detection boxes in the first view and the hand estimation boxes in the second view, redundant boxes corresponding to redundant hands, and then performing hand joint point recognition on remaining boxes to obtain two-dimensional joint points; converting the two-dimensional joint points into three-dimensional joint points in a three-dimensional hand coordinate system; and converting the three-dimensional joint points in the three-dimensional hand coordinate system into three-dimensional joint points of the current frame in a world coordinate system according to pose estimation parameters corresponding to the current frame.

The invention discloses a hand posture recognition method, a head-mounted display device and a storage medium, and relates to the field of man-machine interaction. In the method, projection correction is carried out on an image of a first hand region in a current frame image by iterating a projection direction for multiple times; the present invention corrects an image captured due to inclination of an observation line of sight into an image in which the line of sight is perpendicular to the center. In this way, the hand image sent into model reasoning overcomes the distortion influence, and the positions and the relative distances of the hand joint points obtained through model reasoning are more accurate. According to the invention, the problem that the coordinates of the predicted 3D hand articulation points are inaccurate due to image distortion is solved.

The invention relates to robot motion control, in particular to a dexterous hand motion algorithm control method, system and device, a storage medium and a program product. The dexterous hand motion algorithm control method comprises the following steps: visual detection: acquiring a hand image through a camera, and identifying three-dimensional coordinates of key points of hand joints by adopting a visual detection algorithm; kinematics calculation: based on the coordinates of the joint key points, obtaining target execution angles of a plurality of motors through kinematics forward and inverse calculation; communication distribution: receiving the target execution angle through a master control unit, and distributing the target execution angle to a plurality of slave control units according to a preset distribution rule; and motor control: each slave control unit generates a driving signal by adopting a PID control algorithm based on the received target execution angle, and controls the corresponding motor to move. The dexterous hand motion algorithm control method provided by the invention is high in response speed, high in control precision, stable in communication and accurate in kinematics solution.

The invention provides a driving method of a 3D gesture model and electronic equipment. The driving method is used for improving the driving accuracy of the 3D gesture model. Comprising the following steps: obtaining predicted attitude data of each hand joint point in a current frame of hand image based on attitude data of each hand joint point in a specified frame number of hand images before the current frame of hand image, and obtaining a hand time sequence diagram; obtaining intermediate attitude data of each hand joint point by using a double-time-sequence model based on a time sequence feature graph obtained by performing feature extraction on a target region in the hand time sequence graph and a hand feature graph obtained by performing feature extraction on the current frame of hand image; if the intermediate posture data of the specified hand joint point meets a specified condition, inputting the current frame of hand image and a hand time sequence diagram of a target standard hand posture obtained by matching the intermediate posture data with the posture data of each standard hand posture into a double-time-sequence model to obtain target posture data of each hand joint point, and driving the 3D gesture model by using the target attitude data.

The invention discloses a gesture recognition method fusing continuous frame segmentation and spatial features. The gesture recognition method comprises the steps that three-dimensional coordinates of hand joint points are obtained through Mediape; performing dual normalization processing to eliminate size and position differences; extracting point cloud convex hull features, joint point distance features and specific triangularization area combination features; fusing a plurality of types of features and then training a Transform or GCN model; carrying out automatic segmentation on the continuous gesture sequence based on double cosine similarity thresholds; and inputting the segmented stable paragraphs into the model to realize real-time identification. The method provided by the invention has high precision and strong robustness in a complex scene, and is suitable for the fields of man-machine interaction, intelligent control and the like.

The invention provides a hand posture estimation method and device and head-mounted display equipment, and relates to the technical field of man-machine interaction, and the method comprises the steps: determining the pixel position and the relative distance of each hand joint point, and the bending degree of each finger according to a hand binocular image; according to the bending degree of each finger, an optimization variable of the hand posture is determined; according to the optimization variable, the pixel position and the relative distance of each hand joint point and the bending degree of each finger, an optimization function of the hand posture is generated; and performing iterative optimization on the hand posture according to the optimization function to obtain a hand posture estimation result. By combining the pixel position and the relative distance of each hand joint point and the bending degree of the finger, the hand posture is iteratively optimized, so that the estimation accuracy of the hand posture is improved.

A hand posture estimation method based on dual aggregation network enhancement comprises the following steps: generating a hand point cloud from a hand depth image, and inputting the hand depth image and the point cloud into a local coding fusion module to generate fused image features and point cloud features; inputting the fused 3D point cloud features into an initial state generator to initialize a hidden state; inputting the hidden state into a regression module to obtain initial estimation of a joint point; initially estimating and fusing the image features and the point cloud features, and inputting the image features and the point cloud features into a point cloud image consistency aggregation module to generate enhanced point cloud features; the enhanced point cloud feature and initial estimation input resampling module outputs high-dimensional hand joint point features; the enhanced point cloud features, the high-dimensional hand articulation point features and the hidden state are jointly input into a dynamic graph enhanced aggregation module to obtain enhanced high-dimensional articulation point features; and inputting the enhanced high-dimensional joint point features into a regression module to obtain a final hand joint point coordinate position. According to the method, gesture posture estimation under the shielding condition can be improved, and hand joint output is enhanced.

The invention relates to the technical field of gesture recognition interaction, and discloses an intelligent interaction system and method based on gesture recognition. The method comprises the following steps: when an environment meets a condition, synchronously acquiring an image sequence of a gesture and depth distance field data; a hand joint point two-dimensional motion track is extracted from an image, surface deformation fluctuation information is separated from depth data, the two-dimensional motion track and the surface deformation fluctuation information are subjected to space-time registration fusion, a continuous motion track curved surface in a three-dimensional space is constructed, and then geometric topology features and dynamic change features of the continuous motion track curved surface are extracted. And inputting the fusion features into a neural network classifier subjected to incremental learning training, outputting corresponding semantic tags and confidence evaluation values, and mapping to generate a control instruction after verification. According to the method, an accurate three-dimensional dynamic model is constructed through deep fusion of multi-source data, and an incremental learning mechanism is utilized to enable the system to have online adaptive capability, so that the recognition precision of complex gestures and the long-term applicability of the system are improved.

The invention discloses a hand motion mapping method based on a data glove, which comprises the following steps of: judging the type of a task to be performed according to data acquired by the data glove, and selecting different mapping methods. If the current gesture is judged to be the gesture representation, applying a gesture recognition model, if the current gesture exists in a gesture library, applying a standard gesture in the library, and if the current gesture does not exist, applying direct joint angle mapping to four fingers except the thumb, and applying Cartesian space mapping to the thumb. And if the task is judged to be the operation task, the data of the data glove is directly processed by a pre-trained neural network and is directly mapped into a joint angle corresponding to the target manipulator. The training thought of the neural network is that the human hand joint angle represented by the data glove and each joint angle of the target manipulator under the same gesture action are obtained at the same time, a neural network model is constructed, the angle of the data glove serves as input, the corresponding angle of the manipulator serves as output, and training is conducted. Finally, the motion mapping process from the hand to the manipulator is achieved.

The invention provides a hand multi-mode information sensing device and a robot humanoid skill learning method, and the device comprises a multi-mode touch sensor module which is used for sensing a hand multi-mode touch signal; the flexible strain sensor module is used for sensing a hand joint bending signal; the multi-modal motion capture module is used for sensing hand motion and posture signals, and the hand motion and posture signals comprise hand motion signals and / or hand posture signals; at least one of the hand multi-mode touch signals, the hand joint bending signals and the hand movement and posture signals forms hand multi-mode information representing gesture actions, hand movement and hand touch. According to the invention, sensing of hand multi-modal information in a hand operation process can be realized based on the hand multi-modal information sensing device, establishment of a sensing strategy and a control strategy is facilitated, cross-object and cross-task robust skill migration is realized, and self-adaptive operation capability and task success rate of a robot in an actual complex environment are remarkably improved.

The present invention relates to the technical field of minimally invasive instrument control, and in particular, to a minimally invasive instrument control system and method for surgical operating rooms. The system comprises a master hand operation module, an operation position monitoring module and a tactile feedback module. The present invention controls the contact between a tactile operating arm and a lesion through a tactile feedback module, obtains contact feedback, and sends the contact feedback to the master hand operation module, controls the vibration feedback of a vibration instrument built into the master hand operation module, and feeds back vibrations in real time to the hand through a hand joint encoder held by a doctor by referring to the vibration mode matched with the lesion area, determines the hardness of the lesion area currently contacted by the tactile operating arm, and cooperates with the intraoperative patient lesion image displayed in real time by a 3D image real-time display module to predict the state of the lesion operation point, provide tactile feedback for doctors performing remote surgery, further improve the accuracy of the surgery, and assist doctors in performing surgical operations.

1. The name of the design product: adjustable eating fork. 2. The use of the design product: designed for the general silver-haired population or the elderly, the disabled, and people whose hand joint flexion function is limited, whose grip function is not in place, whose grip strength is insufficient, and whose hands are shaky. 3. The design points of the design product: in shape. 4. The picture or photo that best indicates the design points: perspective view.

The invention discloses a human motion classification method, device and equipment based on RFID and a medium, and the method comprises the steps: collecting human skeleton phase data, and carrying out the preprocessing of the data, and forming RFID skeleton activity data; constructing a human body action diagram, and connecting hand joint nodes and foot joint nodes in the human body action diagram to form an initial human body skeleton structure diagram; and performing graph convolution processing on the initial human body skeleton structure graph to generate a human body action prediction classification label. By adding an improved graph structure formed by connection between the hands and the feet in the human body structure graph, the topological distance between the nodes is reduced, the nodes can be aggregated to more neighbor node information, the aggregation capability of joint features is improved, and the joint feature aggregation efficiency is improved. And during action recognition of cooperation of both hands and hands and feet, richer action feature information can be captured through aggregation features among non-physically connected joints, and the recognition precision of the model on complex postures is improved.

The embodiment of the invention provides a dexterous hand joint angle determination method and equipment. The method comprises the steps that power output parameters of a driving motor of a target dexterous hand are obtained, the driving displacement amount of the target dexterous hand is determined according to the power output parameters, the joint angle of the target dexterous hand is determined according to the driving displacement amount and a first corresponding relation, and the first corresponding relation is determined based on the geometric structure of the target dexterous hand. The first corresponding relation is used for representing the incidence relation between the driving displacement and the joint angle of the target dexterous hand. According to the method, the driving displacement is determined, and the joint angle of the target dexterous hand is determined based on the first corresponding relation between the driving displacement constructed through the geometric structure of the target dexterous hand and the joint angle of the target dexterous hand, so that the determination precision of the joint angle is improved, additional sensors do not need to be installed at the joints, and the accuracy of the joint angle determination is improved. The space is saved.

The embodiment of the invention provides a mechanical finger joint miniature encoder mounting structure for improving control precision, and belongs to the technical field of medical instrument robot dexterous hands and high-precision sensing. The structure comprises a hollow annular magnetic encoder, a customized mounting base and an error compensation mechanism. The MPT-8-20 series encoder is directly integrated on a rotating shaft of a finger joint through the customized mounting base; the mounting base is provided with a positioning interface which is precisely matched with an encoder shell and a joint part, so that the concentricity and a constant air gap of an encoder sensing unit and a rotating part are ensured; the error compensation mechanism is integrated in the base and is used for passively inhibiting installation errors and environmental interference. According to the invention, the measurement precision of less than 0.01 degree in an extremely limited space is realized based on a commercial encoder, high anti-interference performance and reliability are realized, the integration problem of high-precision and high-reliability pose feedback of the miniature dexterous hand joint is effectively solved, and the miniature dexterous hand joint is suitable for miniature dexterous hands.