204 results about "Hand motion" patented technology

The invention discloses a sign language recognition method and device based on multi-modal deep learning, and the method comprises the steps: multi-modal data input: capturing hand motions, gesture tracks and facial expressions at the same time through a camera, a motion capture sensor and other devices, and forming multi-modal data input; sign language action recognition: precise recognition of sign language actions is realized through a combined model of a deep convolutional neural network and a long-short-term memory network; facial expression and gesture track combined recognition: realizing understanding and translation of complex sign language sentences by combining the captured facial expressions and gesture tracks; and context natural language processing: generating a target statement in combination with context semantic understanding, and outputting a translation result. Through hand motion capture, facial expression analysis, gesture trajectory tracking and context natural language processing, complex sign language motions can be recognized more accurately and translated into characters or voices in real time, and low delay and high accuracy are achieved.

The invention provides a self-supervised continuous 3D hand posture tracking method based on a lightweight inertial measurement unit, and the method specifically comprises the following steps: firstly, obtaining a hand motion signal by using a lightweight consumer-level IMU, and extracting features by using a multi-stage neural network containing a bidirectional long-short term memory (Bi-LSTM) module; the global displacement and orientation of the hand are processed through a wrist posture estimation module, and three-dimensional coordinates of hand skeleton points are solved and calculated by means of a finger motion chain; secondly, a self-supervised completion network is realized by combining a random mask and dual loss, and the features are processed to complete complete attitude reconstruction; and finally, combining network output with kinematics prior, generating a three-dimensional grid with reasonable anatomy, solving the problem that joint deformation is inconsistent with torque, and realizing accurate acquisition of a three-dimensional hand posture. For a dynamic hand motion tracking task, the method can adapt to a sparse IMU deployment scene and realize continuous high-precision tracking; the real-time performance demand of daily interaction can be met, and a low-cost solution can be provided for the fields such as medical rehabilitation and virtual interaction which have strict requirements on attitude precision.

The invention discloses a generative interaction method and device for medium-free aerial imaging, and relates to the technical field of man-machine interaction for medium-free aerial imaging. The method comprises the steps of obtaining a reference description text, hand motion data and hand posture data corresponding to a target user, wherein the hand posture data is used for indicating a final posture of hand motion; inputting the hand motion data and the hand posture data into a motion text generation model, outputting a motion text, inputting the reference description text and the motion text into a text fusion model, and outputting a target description text; the target description text is input into a model generation network, a three-dimensional model is output, and the three-dimensional model comprises at least one entity model and at least one environment scene model corresponding to the entity model; and mapping the three-dimensional model to a medium-free aerial imaging device to obtain an aerial suspension image. The problems of low aerial imaging efficiency, lack of dynamic interaction capability and the like of the existing aerial imaging scheme are solved.

The invention provides a digital twinning-combined dexterous hand teleoperation hand action migration method. The method comprises the following steps: acquiring an image sequence of a hand through a binocular camera; for each group of images in the image sequence, determining a three-dimensional coordinate sequence of the hand feature points; based on a pre-established digital twin skeleton model of the dexterous hand, establishing a mapping relation between the hand feature points and corresponding points in the digital twin skeleton model; converting the three-dimensional coordinate sequence of the hand feature point into a three-dimensional coordinate sequence of a corresponding mapping point in the digital twin skeleton model based on the mapping relation, taking the three-dimensional coordinate sequence as the position of the mapped hand feature point, and determining an optimal buckling angle and an optimal abduction angle; and based on the optimal buckling angle and the optimal abduction angle, a joint angle sequence is obtained, and corresponding joint actions in the dexterous hand are controlled to enable the dexterous hand to perform action migration according to hand actions in the image sequence, so that the smoothness and precision of the actions of the dexterous hand in the action migration process are improved.

The invention discloses a bionic finger, a hand motion control device, a bionic hand control method and a robot, and belongs to the technical field of bionic robotics.The device comprises a plurality of flexible electric actuators which comprise electric actuating materials capable of actively deforming under excitation of an electric field; one end of the flexible electric actuator serves as a fixed end to be connected to the finger adaptation base, and the other end of the flexible electric actuator serves as a free end to be connected to the knuckle structure through a force transmission path. Wherein each flexible electric actuator can be independently controlled, and the plurality of flexible electric actuators cooperatively drive the knuckle structure to generate bionic motion; the passive constraint structure is configured to provide anisotropic mechanical constraint for active deformation of the flexible electric actuator, so that the active deformation is guided and converted into driving displacement or driving force in the first preset direction; and the guide structure is used for guiding the deformation direction of the flexible electric actuator. The device can integrate a strain sensor, an angle sensor and a touch sensor, and realizes closed-loop cooperative control through a controller.

The invention relates to the technical field of dexterous hand motion planning, in particular to a dexterous hand motion planning method and system based on deep learning, and the method comprises the steps: obtaining the state information of a dexterous hand, carrying out the multi-modal feature fusion to generate environment perception information, carrying out the dynamic obstacle detection and cooperative signal analysis, generating an obstacle avoidance path, and optimizing the motion path planning. Real-time data are collected through the sensor module, multi-modal features are fused, and environment sensing information is generated to support motion planning of the dexterous hand; collaborative decision-making among multiple dexterous hands is realized by utilizing an internal communication module, and the adaptive ability and task execution efficiency in a complex dynamic environment are remarkably improved. In addition, dynamic obstacle information is shared by prompting a cooperation signal, conflicts are avoided, and the response speed is increased. The method is suitable for efficient motion planning of the dexterous hand in a complex dynamic scene.

The invention discloses a soft reset type electric piano counterweight keyboard, and relates to the technical field of piano equipment. A key assembly is arranged at the top of the piano shell; a hinge frame is fixedly installed at the bottom in the piano shell through bolts, soft and feedback playing hand feeling is achieved through precise mechanical linkage, meanwhile, the position of an inductance piece is changed by sliding and extruding a clamping bead through a balance weight, and high-precision and high-reliability pressing force detection is achieved in combination with optical sensing auxiliary monitoring. The force signal is converted into keyboard'post-touch layer 'control, and a single-key or global tenuto mode is intelligently distributed; in addition, the key contact state is accurately captured through photoelectric induction, space capture is carried out on hand actions in combination with an infrared camera and a sensing radar, and real-time tactile feedback is provided through electromagnetic driving.

The invention relates to the technical field of intelligent control, and discloses a dexterous hand motion planning and control method based on a visual language motion model, and the method comprises the steps: S1, collecting data, and carrying out the time sequence synchronization and consistency verification of multi-source data; s2, performing dynamic adaptive exponential moving average filtering and calculation processing on the joint sequence, and reconstructing a processing result into adaptive structured features; s3, fusing joint states, inputting the fused joint states into an action head network, splicing online and offline samples into a training batch according to a preset proportion, and determining results to jointly optimize model parameters; s4, performing model reasoning to obtain target action information, performing processing in sequence to generate a smooth low-jitter control sequence, and sending the smooth low-jitter control sequence to the dexterous hand for execution; and S5, the deployment thread issues a control instruction by aligning the annular buffer and the timestamp, triggers a rollback track and allows manual intervention when the control instruction exceeds a threshold value, and writes manual intervention data into an offline buffer. According to the method, efficient, stable and smooth motion control of the dexterous hand can be realized under the condition of extremely few teaching data sets.

The invention discloses a piano teaching interaction method based on a skeleton point image recognition technology. The method comprises the following steps: S1, collecting video data of hand actions when a user plays a piano; s2, carrying out preprocessing on the video data; s3, extracting hand skeleton point data by using the skeleton point extraction model to obtain a skeleton key point data set; s4, training a hand action recognition network model by using the extracted skeleton key point data set and the loss function; and S5, inputting the real-time output of the skeleton point extraction model into the trained hand action recognition network model, and detecting the correctness of the hand posture. According to the method, the motion information of the hands of the piano player can be captured and analyzed in real time with high precision, and targeted and personalized teaching feedback can be realized through deep mining of skeleton point data, so that the interactivity and teaching efficiency of piano teaching are improved.

The invention relates to robot motion control, in particular to a dexterous hand motion algorithm control method, system and device, a storage medium and a program product. The dexterous hand motion algorithm control method comprises the following steps: visual detection: acquiring a hand image through a camera, and identifying three-dimensional coordinates of key points of hand joints by adopting a visual detection algorithm; kinematics calculation: based on the coordinates of the joint key points, obtaining target execution angles of a plurality of motors through kinematics forward and inverse calculation; communication distribution: receiving the target execution angle through a master control unit, and distributing the target execution angle to a plurality of slave control units according to a preset distribution rule; and motor control: each slave control unit generates a driving signal by adopting a PID control algorithm based on the received target execution angle, and controls the corresponding motor to move. The dexterous hand motion algorithm control method provided by the invention is high in response speed, high in control precision, stable in communication and accurate in kinematics solution.

The invention relates to the related technical field of student behavior monitoring, in particular to a classroom student behavior monitoring system, which comprises the following functional modules: an identity label generation module; an image acquisition module; a data storage module; a behavior analysis module; a multi-modal data integration module; and an interaction feedback module. According to the classroom student behavior monitoring system, the identity label generation module, the image acquisition module, the data storage module, the behavior analysis module, the multi-modal data integration module and the interaction feedback module work cooperatively; algorithm design of a head posture analysis unit, a hand action recognition unit, an electronic equipment use detection unit and a sitting posture evaluation unit is further optimized, and a student behavior track diagram and an abnormal condition report are provided as dynamic monitoring results to an external user by means of an interaction feedback module. The defects of the prior art in the aspects of recognition precision, real-time performance, multi-dimensional data analysis and privacy protection are overcome.

The invention discloses a virtual sensory integration training method based on force feedback, and the method comprises the steps: constructing a hot chamber virtual environment containing a mechanical arm model and an operation object through a scene simulation platform, and configuring a joint position according to a real mechanical arm parameter; the dynamics simulation platform builds a corresponding dynamics model and configures inverse kinematics solution and collision detection functions; force feedback equipment is integrated, and a working space mapping model is constructed to convert hand motion into the expected pose of the tail end of the mechanical arm; the double platforms synchronize data through a dynamic link library, and a force sense calculation module calculates collision force and bolt assembling and disassembling torque and transmits the collision force and the bolt assembling and disassembling torque to force feedback equipment. The method does not need to depend on a real mechanical arm, safety risks and high cost are avoided, training repeatability and pertinence are improved, immersion and operation precision are enhanced through force sense and visual sense synchronous feedback, the training period is shortened, and the method is suitable for fine operation training of the mechanical arm in a dangerous environment.

The invention discloses a force feedback teleoperation control system and method based on a continuum surgical actuator, and belongs to the technical field of medical surgical robots. The system comprises a master end driving part and a slave end following operation part; the master end driving part is used for acquiring hand action data of an operator in real time, converting the acquired hand action data into a corresponding control signal, and sending the control signal to the slave end following operation part through a preset communication protocol; the slave end following operation part is used for generating a corresponding driving signal and transmitting the generated driving signal to the servo motor so as to control the operation actuator to move; and in the movement process of the operation executor, the interaction force of the operation executor and the target object is sensed in real time, and the interaction force of the operation executor and the target object is fed back to the main end driving part in real time so that an operator can sense the interaction force. According to the scheme, the teleoperation precision can be guaranteed, and meanwhile the telepresence of a doctor is enhanced.

The invention discloses a brain-computer interface manipulator motion detection system for disturbance of consciousness assessment, which belongs to the technical field of disturbance of consciousness assessment and specifically comprises an electroencephalogram acquisition device for acquiring original electroencephalogram signals when a patient executes motor imagery tasks containing different hand action instructions; the dynamic feature extraction module generates a motor imagery activation map by calculating the energy distribution change of a preset frequency band; the driving instruction generation module calculates the space-time matching degree of the map and a preset action instruction and generates a manipulator driving instruction; the manipulator control module is used for controlling and executing a composite action positively correlated with the instruction strength, and the action is formed by combining basic grasping and multi-joint coordination actions in proportion; the motion capture device records track changes, and the ratio of the number of the joints exceeding the threshold value to the motion duration serves as a consciousness activity index; the disturbance of consciousness grading module determines a grading result according to the fluctuation range of the index in the continuous period; the method provides an objective and accurate quantification means for disturbance of consciousness assessment.

The invention relates to the technical field of intelligent wearable equipment, and provides a hand motion capture data glove based on an IMU sensor. The thumb mechanism comprises a first thumb section, a second thumb section, a metacarpal bone shell and a connecting piece, the first thumb section and the second thumb section are connected in a sliding and rotating mode, and the metacarpal bone shell is connected with the main control board through the connecting piece; each finger mechanism comprises a first knuckle, a second knuckle, a third knuckle and a telescopic rotating connecting assembly, the first knuckles are in sliding rotating connection with the second knuckles, the second knuckles are in sliding rotating connection with the third knuckles, and the third knuckles are connected with the main control panel through the telescopic rotating connecting assemblies; the first thumb section, the second thumb section, the metacarpal bone shell, the first knuckle, the second knuckle and the third knuckle are each provided with an IMU sensor, each IMU sensor is electrically connected with the main control board, and the main control board is integrated with the IMU sensors. Therefore, production is simple, and cost is low.

The invention discloses a gesture recognition method, device and equipment based on multiple sensors, a medium and a program product, and the method comprises the steps: decomposing multi-sensor data into a plurality of action frames according to a time sequence based on sensor timestamps, and generating an original sequence data set based on the plurality of action frames; the original sequence data set is preprocessed, a hand action time sequence is generated, and preprocessing comprises the step of compensating the original sequence data set based on the environment data; and distributing data weights of the sensors based on the contribution degrees of the sensors, performing feature fusion on the multi-modal hand action data of the time points in the hand action time sequence based on the data weights, performing time-space coding on the fused features, and inputting time-space coding vectors into a gesture recognition model for gesture recognition. According to the method, effective fusion of multi-modal sensor data is realized, and data compensation is performed by combining environmental data, so that the noise influence of the environment on the sensor is effectively reduced, and the gesture recognition precision is greatly improved.

The invention relates to the technical field of artificial intelligence, and discloses a digital human driving method based on artificial intelligence, comprising the following steps: step 1, acquiring a voice signal, a facial electromyographic signal, a pupil size signal and a hand motion signal of a student through a wearable device, the wearable device comprising a helmet built-in sensor and a hand motion capture device; step 2, converting the voice signal in the step 1 into a fundamental frequency feature vector, converting the facial electromyographic signal into a 28-dimensional muscle activation degree vector, converting the pupil size signal into a diameter change rate scalar, and converting the hand motion signal into a six-degree-of-freedom pose matrix; according to the method, the multi-mode behavior recognition engine is adopted to fuse micro expression analysis, voice stress detection and skeleton motion capture technologies, the technical effect of recognizing the tiny abnormal reaction of the student in real time is achieved, and compared with a visual behavior recognition scheme based on a preset script in the prior art, the defects that deviation correction response is delayed and the safety knowledge memory effect is poor are overcome.

The invention provides a fine motor imagery method and system based on hand motion parameter variation, the fine motor imagery method based on hand motion parameter variation can be applied to a cerebral apoplexy rehabilitation experiment / training system, the system comprises an electroencephalogram acquisition device, a motion capture device, a first computer, a second computer and a bearing platform, the fine motion imagination method based on the hand motion parameter change comprises the following steps: S1, building a high-precision motion capture and electroencephalogram synchronous acquisition experiment platform; S2, carrying out a speed-single parameter fine motion task; and S3, carrying out a speed-direction double-vector parameter fine motion task. Based on the fine motion tasks of different motion parameters, the degree of freedom of the motion imagination task is increased, the available normal form of the motion imagination task is expanded, and the brain coding and decoding rule of the vector space motion intention rich in direction, speed and other information can be further understood by adopting the speed-direction fine motion intention experimental design scheme.

The invention discloses a hand motion mapping method based on a data glove, which comprises the following steps of: judging the type of a task to be performed according to data acquired by the data glove, and selecting different mapping methods. If the current gesture is judged to be the gesture representation, applying a gesture recognition model, if the current gesture exists in a gesture library, applying a standard gesture in the library, and if the current gesture does not exist, applying direct joint angle mapping to four fingers except the thumb, and applying Cartesian space mapping to the thumb. And if the task is judged to be the operation task, the data of the data glove is directly processed by a pre-trained neural network and is directly mapped into a joint angle corresponding to the target manipulator. The training thought of the neural network is that the human hand joint angle represented by the data glove and each joint angle of the target manipulator under the same gesture action are obtained at the same time, a neural network model is constructed, the angle of the data glove serves as input, the corresponding angle of the manipulator serves as output, and training is conducted. Finally, the motion mapping process from the hand to the manipulator is achieved.

The invention provides a hand multi-mode information sensing device and a robot humanoid skill learning method, and the device comprises a multi-mode touch sensor module which is used for sensing a hand multi-mode touch signal; the flexible strain sensor module is used for sensing a hand joint bending signal; the multi-modal motion capture module is used for sensing hand motion and posture signals, and the hand motion and posture signals comprise hand motion signals and / or hand posture signals; at least one of the hand multi-mode touch signals, the hand joint bending signals and the hand movement and posture signals forms hand multi-mode information representing gesture actions, hand movement and hand touch. According to the invention, sensing of hand multi-modal information in a hand operation process can be realized based on the hand multi-modal information sensing device, establishment of a sensing strategy and a control strategy is facilitated, cross-object and cross-task robust skill migration is realized, and self-adaptive operation capability and task success rate of a robot in an actual complex environment are remarkably improved.

This invention discloses an adaptive impedance control method and device for a teleoperated robot system. The method is based on a fuzzy logic-based intelligent control mechanism. It dynamically generates a correction update rate for online compensation adaptive terms through fuzzy inference, based on real-time acquired master-slave state information and preset control parameters. This correction update rate, real-time state information, and preset parameters are integrated, and the core dynamic equations of the preset impedance control model are solved to calculate the robot's target position within the current control cycle. This method, based on the robot following the operator's hand movements, superimposes a position compensation amount optimized by the adaptive impedance control model. When problems such as operator hand tremor or network transmission delay cause a sudden increase in contact force, making it impossible to effectively track the desired contact force, the adaptive impedance control device adjusts the contact force, achieving refined teleoperation control.

The application discloses a kind of based on visual perception nimble hand motion planning method, equipment and medium, it is related to robot control technical field, including, based on the multi-modal visual data of real-time acquisition execution object instance segmentation and object six degree of freedom pose estimation, generate the environment state representation including object category, segmentation mask and pose information;When arbitration decision instruction comprehensive risk difference index does not satisfy preset stable execution condition, start global re-planning and generate new planning instruction, and continuously update environment state representation in executable main trajectory execution process.The application is estimated to object six degree of freedom pose by covariance analysis to result and applies counterfactual disturbance, forms the multi-state environment representation around the perception uncertainty in trajectory planning phase, realizes multi-trajectory planning expression under the same task instruction constraint.

The utility model discloses a flexible film motor providing tactile feedback, a multilayer film motor and a glove, the flexible film motor comprises a first film layer, a support layer and a second film layer, the film layer comprises an insulation substrate, a flexible electrode and a dielectric layer, the insulation substrate attached to the skin is arranged on one side of the flexible electrode, and the dielectric layer is arranged on the other side of the flexible electrode. The dielectric layer is arranged on one surface, far away from the insulating substrate, of the flexible electrode, and the supporting layer is arranged between the dielectric layer of the first thin film layer and the dielectric layer of the second thin film layer. The flexible thin-film motor provided by the embodiment of the utility model can generate mechanical vibration through electrostatic force, thereby generating efficient tactile feedback, and is wide in operation frequency domain, high in feedback precision, and high in reliability. Meanwhile, it can be guaranteed that the tactile feedback glove can keep good fitness and operation flexibility when carrying out various hand actions, and a user can carry out fine operation and natural hand movement.

The invention discloses a virtual seven-stringed plucked instrument practice system, belongs to the technical field of virtual simulation training, and solves the technical problems of sensing error interference and lack of pertinence and individuation of guidance in existing virtual seven-stringed plucked instrument practice. The system comprises a data acquisition module, a data processing module and a feedback output module. Hand motion sensing data are collected through a data glove, consistency deviation generated by equipment factors and posture deviation generated by user operation are extracted, user types are divided by combining the difference degree of a playing effect and a standard effect and posture deviation characteristics, consistency deviation compensation or compensation and posture correction is executed in a classified mode, and the user performance is improved. Personalized standard hand shape adaptation, dynamic adjustment of a judgment threshold value, deviation pre-judgment and reverse optimization of acquisition sensitivity can also be realized, and a feedback module synchronously outputs virtual images and sound feedback. According to the invention, the accuracy of the sensing data is improved, personalized and real-time Guqin practice guidance is realized, and the Guqin practice experience and practice efficiency are effectively optimized.

The invention discloses a dexterous hand motion capture system and method and a readable storage medium, and belongs to the technical field of motion capture. The motion capture system comprises a dexterous hand motion capture sensor group, a dexterous hand reference template creation unit and a dexterous hand calculation unit, wherein the dexterous hand motion capture sensor group comprises a near-infrared or infrared band image sensor and a visible light band image sensor; the dexterous hand reference template creating unit is used for outputting a dexterous hand reference template based on the dexterous hand image in the near-infrared or infrared band in the reference state; and the dexterous hand calculation unit is used for processing output data of the dexterous hand motion capture sensor group based on the dexterous hand reference template and determining motion data of the dexterous hand. According to the invention, the motion data of the dexterous hand is obtained based on two different modes of dexterous hand images, namely, the near-infrared or infrared band image and the visible light band image, so that the precision and robustness of capturing the motion of the dexterous hand by a dexterous hand motion capturing system are improved.

The invention discloses an air bag brake mechanism, and particularly relates to furniture, tables and chairs and other related technical fields, the air bag brake mechanism comprises an auxiliary brake mechanism, the auxiliary brake mechanism comprises an upper fixing plate, a lower fixing plate, a brake plate, a brake connecting shaft, a connecting bolt and a wheelchair structure, the bottom of the upper fixing plate is provided with a plurality of sets of supporting columns, and the supporting columns are sleeved with movable plates; a locking bolt is arranged at the top of each supporting column; wherein the upper fixing plate and the lower fixing plate are connected through the supporting columns and the locking bolts to form a fixed cage body; the auxiliary braking mechanism is arranged, the manual inflation bag designed according to the human engineering principle is innovatively adopted in the auxiliary braking mechanism, and a user can easily achieve braking and braking releasing of the table and the chair only through the simple action of manual squeezing and releasing; according to the auxiliary braking mechanism, a user does not need to stoop to operate and only needs to naturally carry out simple hand actions by hands beside a table and a chair, and the operation process is greatly simplified through the design.

The invention relates to the technical field of simulation, in particular to a finger motion trail simulation method and device, equipment, a medium and a program product, and the method comprises the steps: obtaining a hand skeleton image and a hand motion video of a detected target; constructing a three-dimensional hand simulation model according to the hand skeleton image; extracting a plurality of video frames from the hand action video, extracting a two-dimensional coordinate of at least one position point from each of the plurality of video frames, converting the hand action video into a three-dimensional coordinate based on a pre-established coordinate mapping relationship, and forming time sequence data according to the position point of the three-dimensional coordinate; and driving the three-dimensional hand simulation model to execute a simulation action by using the time sequence data, and generating a finger motion track of the tested target according to an execution result of the simulation action. Therefore, the problems of high acquisition cost, poor universality and the like of the finger motion trail in the prior art are solved.

The invention relates to the technical field of action recognition, and provides a hand action recognition method and system based on a lightweight cross-modal space-time network, and the method comprises the following steps: carrying out the preprocessing of a hand-washing video stream, and obtaining an RGB image sequence and a hand key point coordinate sequence; performing separation convolution operation of multi-layer space convolution and time convolution on the RGB image sequence, extracting motion-related features in combination with a channel attention mechanism, and generating a visual feature mark sequence; performing semantic extraction and time sequence compression on the key point coordinate sequence to obtain a structural feature mark sequence aligned with the X; and dynamically calculating a gating coefficient based on context information, fusing X and H to obtain fusion features, and outputting classification probability distribution of the hand washing steps. According to the method, a lightweight visual feature extraction structure is adopted, and a dynamic gating mechanism is introduced to realize adaptive collaborative fusion of visual and structural modes, so that high-precision and low-delay hand action recognition is realized on edge equipment.