24 results about "Gesture tracking" patented technology

The application provides a female pelvic cavity simulation module, a VR simulation training system and method for pelvic cavity examination. The VR simulation training system comprises a female pelvic cavity simulation module, a speculum, a gesture tracking device, a mobile locator and VR glasses connected to the mobile locator. The female pelvic cavity simulation module comprises a shell, a suspension assembly, a vaginal simulation assembly, a sensing assembly and a controller. The shell has a receiving cavity. The vaginal simulation assembly is suspended in the receiving cavity through the suspension assembly. The vaginal simulation assembly comprises two simulation bodies arranged in an up-down manner. The simulation bodies are made of soft elastic material. The simulation bodies form a vaginal cavity with an open end when fitted. The sensing assembly is used to detect the moving distance of the two simulation bodies. The controller is electrically connected to the sensing assembly and located above the mobile locator. The application simulates a real clinic environment by using VR. In combination with a virtual character and a user interface, appropriate feedback is given to enhance the students' confidence and adaptability in gynecological examination training.

The application provides a writing pen touch recognition method based on gesture recognition and inertial sensors, relates to the touch recognition technical field, and is applied to a gesture-pen fusion positioning system. Inertial sensing device data is acquired and processed into the positioning coordinates of a writing pen nib in a three-dimensional space, then into the height information of the pen nib to a writing plane, and finally into the touch recognition result through linkage with the vibration information of the inertial sensing device. The application utilizes the gesture tracking capability commonly used in VR / MR / AR devices on the market, avoids customizing the writing pen, can be embedded in the existing devices at a low cost, is not limited to writing surfaces and writing pen types, has high universality, recognizes touch through the pen nib height auxiliary vibration intensity, and makes the recognition result more accurate and reliable.

There is provided techniques for gesture recognition in a gesture tracking application. A method is performed by a controller device. The method comprises receiving user input from a user of the gesture tracking application. The user input identifies a set of gestures for the gesture tracking application. The method comprises training a BSMGRS and a CAMGRS to independently of each other recognize the set of gestures for the user. The method comprises establishing a database of gestures for the user. The database of gestures comprises a set of approved gestures comprises those of the gestures in the set of gestures for which the training of both the BSMGRS and the CAMGRS was successful. The method comprises performing gesture recognition of the user for the gesture tracking application. The gesture recognition is performed using the BSMGRS and in accordance with the established database of gestures.

The invention discloses a gesture tracking method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the time synchronization of an obtained hand motion data set and an obtained electromyographic signal data set based on a preset optical signal, and obtaining the synchronized hand motion data set and the synchronized electromyographic signal data set; associating the hand action data and the electromyographic signal data at the same moment in the synchronized hand action data set and the electromyographic signal data set to construct a plurality of training samples; according to each training sample, performing iterative training on a preset to-be-trained model to obtain a gesture tracking model; and tracking the gesture of the target user based on the gesture tracking model. According to the method and the device, the natural interaction experience between the user and the equipment can be improved while the gesture tracking precision is ensured.

The invention discloses a copying gesture tracking system for an intelligent learning desk, which comprises an overturning plate, the overturning plate is arranged on the desktop of the intelligent learning desk, the upper surface and the lower surface of the overturning plate are respectively provided with an upper sinking groove and a lower sinking groove, and the upper sinking groove is provided with an adaptive transparent plate; comprising a conveying unit, the conveying unit comprises two annular air belts symmetrically distributed on the left side and the right side of an upper sinking groove and two driving wheels corresponding to the annular air belts, the front side wall and the rear side wall of the upper sinking groove are each provided with an arc-shaped channel used for being communicated with a lower sinking groove, and one driving wheel is rotationally connected into each arc-shaped channel; and each annular air belt is arranged on two opposite driving wheels in a sleeving manner. According to the invention, the copying learning can be completed only through small-range rotation of eyeballs without neck movement, the sight line does not need to be repositioned in the whole process, the best state of copying and reading can be maintained, namely, the user can enter a concentration state more easily, so that the copying efficiency is improved.

This invention discloses a high-precision gesture tracking method and system based on hand and controller fusion, belonging to the field of virtual reality and augmented reality technology. It addresses the technical problems in existing VR / AR systems, such as unknown extrinsic parameter estimation for hands and controllers, difficulty in tightly coupling low-frequency vision with high-frequency IMU, and self-occlusion issues in handheld states. The method includes: determining the grip state between the controller and hand based on acquired multi-source data; if the grip state is unoccluded or low-occlusion, performing optimal solution calculations under relevant kinematic constraints to obtain the wrist 6-DoF pose; if the grip state is high-occlusion, performing regression mapping processing on the infrared spot data and controller IMU data based on relevant spatiotemporal feature sequences; dynamically switching the gesture tracking between the controller and hand, and outputting the final hand posture data.

The invention relates to a method, a device and a system for preventing fingertip pinching mold penetration in VR gesture tracking and application. The method comprises the following steps: acquiring real-time three-dimensional space coordinates of a thumb fingertip joint point and an index finger fingertip joint point, and calculating a distance d between the thumb fingertip joint point and the index finger fingertip joint point; establishing a multi-level threshold group including an approach threshold and a contact threshold, establishing an anti-mold-penetrating preparation mode and an anti-mold-penetrating constraint mode, and triggering a corresponding mode based on the relationship between d and the multi-level threshold group; after the anti-mold-penetration preparation mode is triggered, the three-dimensional space coordinate positions of the thumb fingertip joint point and the index finger fingertip joint point are corrected; driving a virtual hand model to render based on the corrected fingertip position and other uncorrected hand joint point data; the device and the system are implemented based on the method, and are applied to the correction of kneading mold penetration in a VR man-machine gesture interaction process. According to the method, the interaction naturalness and immersion are remarkably improved, stable system performance is ensured, the problem of physical authenticity of micro-interaction is solved, and coherence, smoothness and naturalness of the whole hand posture are ensured.

The application is suitable for the technical field of image processing, and provides a gesture tracking method and device, terminal equipment and computer readable storage medium, the method comprises the following steps: inputting a target image into a gesture tracking network model for processing, and outputting a gesture tracking result; if the gesture tracking result is that the target image contains a gesture, determining a first gesture box corresponding to the target image, and the inside of the first gesture box contains target gesture content. The above method can improve the accuracy of the terminal equipment in tracking the hand. And by performing frame skipping processing on the video image, the processing amount of the terminal equipment in processing the video image can be reduced, and the processing efficiency of the terminal equipment can be improved.

There is provided techniques for switching power modes in a gaze-and-gesture tracking application. A method is performed by a controller device. The method comprises activating a first UI input mode for the gaze-and-gesture tracking application. The first UI input mode is based on a first combination of EOG-based or camera-based eye tracking and EMG-based or camera-based gesture tracking. The method comprises, responsive to detecting that eye tracking and / or gesture recognition performance of the first UI input mode fails to fulfil a performance criterion for the gaze-and-gesture tracking application, activating a second UI input mode for the gaze-and-gesture tracking application. The second UI input mode is based on a second combination of EOG-based or camera-based eye tracking and EMG-based or camera-based gesture tracking, different from the first combination.

The application provides a three-dimensional gesture tracking method based on an RGB camera, which comprises the following steps: taking the RGB camera as an input, performing standardization processing on a picture, and sending the picture into a three-dimensional joint detection module to extract image features, predict a preliminary three-dimensional joint and calculate a hand skeleton length; using a particle swarm optimization algorithm to iteratively fit the hand skeleton length and a MAMO hand model, finding an optimal hand shape and obtaining a new three-dimensional joint; sending the new three-dimensional joint into an inverse kinematics module to solve posture parameters required by the MAMO model; using a MANO model to combine the optimal hand shape and the posture parameters θ to obtain final joints and grid vertices; and performing reconstruction rendering on the obtained grid vertices and joints to output a final three-dimensional hand real-time tracking posture. The method provided by the application has high prediction accuracy and high three-dimensional gesture tracking effect precision, and the tracking process and result are easy to realize.

Provided in the present disclosure are a gesture tracking method and apparatus, and a computer device and a readable storage medium. The method comprises: acquiring a vehicle cabin video data stream, performing image capturing on the vehicle cabin video data stream to obtain several image frames; triggering hand detection on the image frames at preset time intervals, so as to obtain hand position information of each triggered image frame; for each triggered image frame, determining hand position information of each non-triggered image frame on the basis of the hand position information of the triggered image frame; matching hand position information of hands on the basis of hand position information of every two consecutive image frames to obtain a matching result of the hand position information of the hands; and determining a dynamic gesture in the vehicle cabin video data stream on the basis of the matching result of the hand position information of the hands in every two consecutive image frames. By means of the method, the computing power required for gesture tracking is reduced, the requirements for the computing capability of a device are lowered, and the limitations are reduced, thereby ensuring that gesture tracking can be normally performed by means of each device.

The invention relates to a filtering method, device and system for improving kneading action stability in VR gesture tracking and application. A kneading action state classification model is established; real-time three-dimensional coordinates of a thumb fingertip joint point and an index finger fingertip joint point are obtained, the distance between the thumb fingertip joint point and the index finger fingertip joint point is calculated, and kneading action state classification is conducted through the model; on the basis of different kneading action state categories, adaptive filtering processing is executed, and the data of the thumb fingertip joint point and the index finger fingertip joint point after filtering processing is used for driving a virtual hand model; the filtering device and the filtering system are realized based on the method, and are applied to gesture tracking in VR human-computer interaction. According to the method, different stages of the kneading action are accurately identified, and an accurate context basis is provided for switching of filtering strategies; dynamically associating the identified kneading state with the filtering intensity to realize intelligent switching; and unification of calculation efficiency and problem pertinence is realized.

The application relates to the technical field of medical education, in particular to a central venous catheter medical operation training method and system based on virtual reality and gesture tracking, which comprises the following steps: collecting user hand action data in real time, identifying standard central venous catheter operation actions performed by a user based on the hand action data through a hierarchical fusion gesture recognition algorithm, analyzing the current operation intention of the user in combination with virtual reality visual scene information, generating and outputting corresponding multi-modal tactile feedback signals according to the identified operation actions and the current operation intention, quantitatively evaluating the operation of the user through an evaluation engine based on the hand action data, the operation action recognition result and the current operation intention, and generating a comprehensive evaluation report. Through high-precision gesture tracking, intelligent intention understanding, multi-modal tactile feedback and automatic quantitative evaluation, the application realizes high immersion of the training process and objective and accurate evaluation of operation skills, and effectively improves the authenticity, safety and teaching efficiency of medical operation training.

A monocular camera augmented reality gesture interaction method based on key point detection, which is composed of the steps of obtaining an input image, detecting hand key point coordinates, determining string data, transmitting string data S, constructing a virtual hand model, storing string data S, virtual hand movement, triggering augmented reality gesture interaction, and performance evaluation. Since the application adopts the steps of determining string data, storing string data S, virtual hand movement, triggering augmented reality gesture interaction, and using a hand key point detection network, the network is applied to the monocular camera gesture tracking and recognition of augmented reality, effectively filters the features of the current frame hand, and standardizes the tracked hand key point information, so that the data is more standardized and convenient for driving the movement of the virtual world hand, and the real-time performance is high. The application has the advantages of higher real-time performance, stronger immersion, lower device cost, etc., and can perform gesture tracking and recognition in different backgrounds.

The invention relates to a multi-camera gesture tracking method for VR / AR equipment, electronic equipment and a medium, and the method comprises the steps: obtaining a current frame image collected by at least two cameras, and predicting a key node prediction position of a current frame through a motion model based on a key node position of a historical frame; for each camera, generating a corresponding ROI on an image plane of the camera according to the predicted position and the current pose of the key node, selecting an optimal camera as a target camera, performing gesture recognition based on an ROI image of the target camera to obtain a gesture recognition result, and updating the position of the key node; electronic equipment and a medium are realized based on the method. According to the method, high-precision and low-delay wrist root node position prediction based on historical motion states is realized, margins are reserved for subsequent processing, dynamic prediction precision is improved, continuity of a tracking chain is ensured, tracking robustness is enhanced, trajectory smoothness and real-time balance are optimized, and overall identification precision and robustness are improved from an input level.

The invention discloses a VR / AR technology-based virtual operation simulation training system and an interaction method thereof, and relates to the technical field of virtual operation simulation training. The multi-user cross-region immersive collaborative operation training system is constructed, role division and collaboration are supported, and the problem of team training deficiency is solved. Multi-dimensional quantitative evaluation is realized based on the expert knowledge graph, operation compliance, safety and technical key points are covered, and personalized improvement suggestions are provided. And by integrating haptic feedback, gesture tracking and eye movement analysis technologies, clinical-level precision interaction is realized. And VR / AR dual modes are supported, and AR adopts an SLAM technology to realize submillimeter registration. A finite element and mass point-spring hybrid algorithm is adopted to simulate a soft tissue physical process in real time. The case complexity is adaptively adjusted, closed-loop training optimization is formed, and training data recording and multi-dimensional report generation are supported.

The invention discloses a VR scenarized application method based on hybrid multi-modal interaction. According to the method, the operation fluency and the intention recognition accuracy in the VR scene are remarkably improved through mixed multi-modal interaction. The gesture topology pre-judgment algorithm can complement joint positions in shielding or rapid actions, maintain continuity of instrument positioning and avoid operation lagging caused by interruption of single gesture tracking; eye movement data and gesture tracks are combined, so that the system captures a focusing target of a user in advance, an intention recognition module is assisted to more accurately understand user requirements, and misjudgment caused by fuzzy single-mode information is reduced. The cooperation of the voice instruction and the visual action further supplements the details of the interaction intention, so that the complex operation can be quickly responded by the system, the user does not need to repeatedly adjust the action or repeat the instruction, and the operation process is smoother.

There is provided techniques for eye tracking and gesture recognition in a gaze-and-gesture tracking application that combines eye tracking for object selection and gesture recognition for action selection. The gaze-and-gesture tracking application is to be run in an XR device that comprises a display. The method is performed by a controller device. The method comprises activating a first UI input mode for the gaze-and-gesture tracking application based on display-based eye tracking. The display-based eye tracking is performed for detecting user selection of objects as rendered on the display. The method comprises, responsive to detecting a critical low power condition, deactivate the display. The method comprises activating a second UI input mode for the gaze-and-gesture tracking application based on display-less eye tracking and / or gesture tracking. The display-less eye tracking and / or gesture tracking is performed for detecting user selection of objects without any visual feedback.

This application provides a method and system for virtual tactile interaction of cultural relics based on multimodal perception fusion. The method includes: preprocessing a 3D ancient scene and a 3D digital model of the cultural relic corresponding to a physical cultural relic model; transmitting one of the physical cultural relic models of different materials to a designated interactive area according to the progress of content preview; identifying the spatial information of the physical cultural relic model using spatial positioning technology, and fusing the spatial information of the physical cultural relic model with the pose of a virtual camera in a VR headset; activating a gesture tracking device to perform real-time gesture tracking, acquiring real-time gesture tracking data, and rendering a virtual hand in the virtual space based on the real-time gesture tracking data; when a viewer wears VR headsets to observe the virtual scene, while the real hand touches the physical cultural relic model, the virtual hand simultaneously touches the virtual cultural relic model in the virtual space. This application improves the accuracy of virtual-real pose fusion during virtual cultural relic interaction, reduces interaction response latency, and provides a better user experience.

The invention discloses a gesture tracking method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the time synchronization of an obtained hand motion data set and an obtained electromyographic signal data set based on a preset audio signal, and obtaining the synchronized hand motion data set and the synchronized electromyographic signal data set; associating the hand action data and the electromyographic signal data at the same moment in the synchronized hand action data set and the electromyographic signal data set to construct a plurality of training samples; according to each training sample, performing iterative training on a preset to-be-trained model to obtain a gesture tracking model; and tracking the gesture of the target user based on the gesture tracking model. According to the method and the device, the natural interaction experience between the user and the equipment can be improved while the gesture tracking precision is ensured.

The invention discloses a gesture tracking method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the time synchronization of an obtained hand motion data set and an obtained electromyographic signal data set based on at least one reference timestamp of synchronization equipment, obtaining a synchronized hand action data set and an electromyographic signal data set; associating the hand action data and the electromyographic signal data at the same moment in the synchronized hand action data set and the electromyographic signal data set to construct a plurality of training samples; according to each training sample, performing iterative training on a preset to-be-trained model to obtain a gesture tracking model; and tracking the gesture of the target user based on the gesture tracking model. According to the method and the device, the natural interaction experience between the user and the equipment can be improved while the gesture tracking precision is ensured.

The application discloses a kind of intelligent platform control method and system, collection operation instruction input signal, and the input signal is processed by multimodal perception unit, generates trigger signal to complete the switching from manual intervention mode to autonomous operation mode, the multimodal perception unit includes voice recognition module, gesture tracking module and touch detection module;According to the preset operation framework diagram, each functional node in intelligent platform is sequentially executed dynamic monitoring, and the real-time state information of each functional node is recorded;Extract the state parameters of the currently running functional node, and correlate analysis is carried out with the state parameters of other functional nodes, to build node relationship map;Based on the node relationship map, the functional node corresponding to the abnormal state parameter is automatically corrected, and its operating parameter is adjusted to normal range.Through dynamic monitoring to functional node and record real-time state, user can accurately perceive platform running state, effectively reduce the risk of misoperation.

The application relates to the technical field of motion capture technology, and discloses a motion capture system for tracking finger postures by fusing light and inertia, which comprises a light-inertia motion capture glove and a light-inertia fusion tracking module. The glove is provided with one handback light-inertia rigid body and ten finger light-inertia single points, and each of the light-inertia single points is integrated with a 6-axis IMU and an active light marker point with a unique flickering code. The tracking module asynchronously runs an IMU thread, a tracking thread and a motion capture thread. After static initialization and light-inertia coordinate system self-calibration, a tight-coupling light-inertia fusion algorithm is used to solve the postures, and 16 segment skeletons with 31 degrees of freedom are output. The application combines the advantages of optical and inertial sensing, has strong anti-shielding capability and no magnetic field interference, suppresses posture drift and zero offset error through double closed-loop feedback, has a short initialization process time, and is suitable for high-precision finger motion capture in scenes such as VR / AR, embodied intelligence and medical rehabilitation.