25 results about "Touch Perception" patented technology

The application discloses a robot complex scene perception system and method based on environment adaptive multi-modal fusion, relates to the technical field of robot perception, and comprises four modules of a flexible sensing matrix, environment feature self-calibration, cross-modal causal reasoning and scene prediction.The flexible sensing matrix collects environment parameters and multi-modal data of vision, inertia and touch.The self-calibration module corrects data errors through environment-error mapping and gradient descent algorithm.The causal reasoning module realizes data fusion and failure completion based on a Bayesian network.The scene prediction module outputs a prediction result through a pre-trained LSTM model and feeds back an optimized collection strategy.The application breaks through the bottleneck of traditional perception technology, improves the environment adaptability, system fault tolerance and decision initiative of complex unstructured scenes, is suitable for scenes such as disaster rescue and outdoor inspection, and helps robots to work autonomously and safely.

This invention discloses a robot visual-tactile perception method and system based on transformer and contrastive learning, belonging to the field of robotics technology. The method introduces an enhanced Swing Transformer model as the perception model. This enhanced model uses the Swing Transformer as the backbone network and incorporates a coordinate attention mechanism. Global information is extracted through the sliding window attention mechanism of the backbone network, while local feature extraction is enhanced through the coordinate attention mechanism. This effectively extracts and fuses global and local features from visual and tactile data. In the pre-training stage, the enhanced Swing Transformer model undergoes multimodal contrastive learning training, specifically intra-modal and inter-modal contrastive learning, generating richer and more accurate discriminative features, thereby effectively improving the robot's intelligent perception and manipulation capabilities. This method can effectively extract features in multimodal data fusion and utilize the correlation and synergistic effects of features between different modalities to achieve deeper multimodal data fusion.

The application discloses a kind of based on high-density tactile-visual and hearing fusion's guide blind device, including high-density vibration tactile feedback main body, visual perception module, voice interaction module, double platform collaborative control unit, map processing and path planning module, positioning navigation module, power management module and mechanical shell.High-density vibration tactile feedback main body adopts vibration motor array to form two-dimensional tactile output interface;Visual perception module realizes hand position real-time detection and coordinate mapping;Voice interaction module is used for full offline speech recognition and synthesis.Corresponding guide blind method is simultaneously provided.The application converts indoor space layout and navigation path into intuitive tactile signal, significantly improves the information carrying capacity of guide blind information, interactive intuitiveness and system robustness, and provides efficient, safe and intelligent indoor travel assistance for visually impaired people.

This invention discloses a guide device and system that converts image signals into vibration signals, relating to the field of assistive technology for the blind. Its key technical points are: the device and system of this invention convert environmental semantic information into perceptible tactile signals, assisting blind people to more comprehensively understand the location of obstacles, road surface properties, and traffic conditions, while simultaneously avoiding dangers on the road. This invention solves the technical problems of low safety and low user acceptance of existing invasive guide devices, as well as poor real-time information transmission in existing guide devices. Furthermore, the printed circuit board assembly of this invention can be flexibly embedded in portable miniature devices, providing blind users with a more natural and convenient guide device.

The application discloses a kind of based on lateral light guide and spatial perception neural network's visual haptics deformation reconstruction device and its reconstruction method, reconstruction device includes: micro camera;Sleeve is configured to be covered outside micro camera, multiple mounting holes are formed on the barrel wall of sleeve;Light guide plate has reflecting surface;Elastic component has reflective layer;Multiple optical fibers are located in corresponding mounting hole;The light emitted by optical fiber is reflected to elastic component by reflecting surface, and irradiates reflective layer;When elastic component contacts target, reflective layer is deformed, micro camera is configured to take pictures to reflective layer and obtain haptic image, so as to determine the haptic information of target according to haptic image.Multiple optical fibers emit axial light under the action of light guide plate to form multiple lateral light, that is, annular discrete dot array light source is formed, the uniformity of light in elastic component is improved, so as to facilitate to improve the accuracy of three-dimensional deformation reconstruction.

The application provides a monocular three-dimensional visual tactile sensor and a sensing method thereof, comprising an elastic contact layer, a speckle coding layer, a monocular visual module, a light source module and a data processing unit; the application aims at the problems of the existing visual tactile sensor, such as complex structure, signal crosstalk, limited resolution, poor environmental adaptability and shallow modal fusion, and provides a visual tactile sensor and a sensing method thereof with simplified structure, high precision, fast dynamic response and strong environmental robustness through monocular three-dimensional DIC technology, super-elastic constitutive modeling and adaptive feature fusion algorithm, so that the real-time collaborative sensing of multi-dimensional information of a contacted object is realized.

This application relates to the field of robot perception and intelligent operation technology, and discloses an object recognition method based on visual-tactile-text multimodal fusion. The method includes acquiring multimodal input data of the object to be recognized, including visual, tactile, and text modal data; extracting visual, tactile, and text modal feature sequences based on the visual, tactile, and text modal data; employing a parameter-sharing multi-head cross-attention mechanism to obtain six sets of cross-modal fusion features; concatenating two sets of cross-modal fusion features corresponding to each modality in the visual, tactile, and text modal data to obtain enhanced visual, tactile, and text modal feature representations; cascading the enhanced visual, tactile, and text modal feature representations and inputting them into a self-attention module to output multimodal fusion features; and inputting the multimodal fusion features into a classification head network for recognition, outputting object category prediction results. This method can improve the accuracy and robustness of object recognition.

A kind of visual tactile sensor and its multi-object hardness real-time measurement method belong to robot tactile and hardness measurement field.It includes camera support frame, light source, monocular camera and tactile perception module, monocular camera is installed in camera support frame side by support plate, light source is used to illuminate camera support frame inside;Tactile perception module is composed of deformable gel layer, mark layer and reflective protective layer, one side of deformable gel layer is installed in the other side of camera support frame by transparent connecting plate, the distance between deformable gel layer and monocular camera is not less than 2 times camera focal length;The other side of deformable gel layer is marked with random pixel as mark layer, reflective protective layer is attached to the remaining surface of deformable gel layer except the surface opposite to monocular camera.The visual tactile sensor structure presented in the application is simple, wide range, can detect the hardness of multiple objects simultaneously, easy to operate.It is more suitable for perception tasks in complex environments such as robot grasping, dexterous hand operation.

Embodiments of the present application provide a pseudo-haptic weight perception method, device, equipment and storage medium, and relate to the technical field of virtual reality. When it is identified that a target object is selected, the length of an interaction ray is calculated and fixed according to a real-time pose and an initial position of the target object, in each time frame according to a preset time frame interval, a ray ideal end point position is calculated according to the real-time pose and the length of the interaction ray, a dynamics model is established according to a virtual mass, an actual rendering position of the target object is calculated based on the dynamics model, a spatial deviation vector of the ray ideal end point position and the actual rendering position is obtained, a non-linear curved ray is generated and rendered based on the spatial deviation vector, the actual rendering position is applied to the target object or a local anchor point of the target object, and a visual representation of a controller at the real-time pose is maintained. A user distinguishes the difference of different virtual masses through the motion retardation degree of the target object and the curved deformation of the ray, and the accuracy of virtual reality weight perception is improved.

The application discloses a quadruped robot multi-modal perception grasping intention recognition compliant control method and system, and belongs to the technical field of robot control. The system comprises a multi-modal perception end effector and a calculation control unit; the effector is carried on the robot trunk and the wrist, and integrates a visual perception module, a tactile perception module and a robot body perception module; the calculation control unit internally comprises three modules of intention recognition, state estimation and adaptive compliant control; the method collects multi-source perception data through the effector, fuses natural language instructions and visual information through a light-weight graph neural network to recognize a grasping intention and output a parameter package, relies on an EKF filtering algorithm to fuse multi-modal data to complete state estimation such as target pose and grasping stability, and then outputs joint torque instructions through base disturbance feedforward compensation and impedance control law calculation, so that dynamic adaptive compliant grasping is realized, and the full autonomous grasping operation demand of an unstructured environment is adapted.

The application relates to the technical field of auxiliary perception, in particular to an auxiliary system and method for perceiving a target, which comprises the following steps: delimiting a haptic perception field; establishing a target object model of a target to be perceived; when the target to be perceived is a two-dimensional structure, forming a target exploration section and acquiring virtual coordinate information; when the target to be perceived is a three-dimensional structure, forming a target exploration shell layer with a preset thickness and acquiring virtual coordinate information; acquiring position information of a haptic perception part of a user; if the position information is located in the range of the target exploration section or the target exploration shell layer, a signal for perceiving the target is fed back to the user; if the position information is located in the range of the target object model except the target exploration section or the target exploration shell layer, a haptic guiding signal is fed back to the user. Through the target exploration section, the target exploration shell layer and the guiding area, the application can ensure the explorability and reduce the phenomenon of haptic form distortion, so that the user can more accurately understand the two-dimensional image contour and the three-dimensional object structure.

The present invention relates to a method and system for processing a digital representation of a visual environment to control a haptic interface including a waistband having active surfaces of N×M actuators, where N and M are integers greater than or equal to 10, comprising an actuator activation protocol that computes a sequence of P activation frames for these actuators for each acquisition of the visual environment, where P is an integer between 2 and 1000, preferably between 5 and 50, each of these frames corresponding to a representation of the environment in an incremental depth plane, characterized in that the method further comprises a processing step of applying a context correction function that modifies the haptic pixel activation protocol.

The application relates to the technical field of robot perception, and provides a visual-tactile sensor and a tactile reconstruction method.The visual-tactile sensor comprises a partitioned elastic contact layer, an illumination control module and an image acquisition module; the partitioned elastic contact layer comprises a high-sensitivity zone, a transition zone and a high-load-bearing zone; the elastic modulus of the high-sensitivity zone, the transition zone and the high-load-bearing zone increases in turn; the illumination control module is used for providing illumination light, and the image acquisition module is used for acquiring a deformation image of the partitioned elastic contact layer under the illumination light.The application divides the elastic contact layer into the high-sensitivity zone, the transition zone and the high-load-bearing zone, and adopts differentiated design of gradient elastic modulus for the zones, so that different zones can accurately perceive and reflect different degrees of contact force through deformation, thereby giving consideration to sensitivity and range, showing high sensitivity in different stress zones, significantly reducing the probability of deformation saturation, and ensuring the accuracy of the deformation image.

The application provides a synapse-like bimodal sensing device, which integrates a bimodal sensing device in one body, significantly simplifies the structure of the bimodal sensing device, and enables the bimodal sensing device to simulate the timely response and processing of nerve synapses to tactile signals and visual signals, thereby exhibiting the performance characteristics of a synapse-like bimodal sensing device. The synapse-like bimodal sensing device comprises a transparent substrate, a stress layer, a photosensitive layer, and an electrode connected to the photosensitive layer; the stress layer and the photosensitive layer are distributed on two side surfaces of the transparent substrate, wherein the projected area of the stress layer on the transparent substrate is not greater than the projected area of the photosensitive layer on the transparent substrate, and the photosensitive layer comprises a metal oxide layer.

The application provides a dexterous hand force-position hybrid control method based on tactile-visual fusion, a robot and a medium, and belongs to the technical field of robot control. The method comprises the following steps: determining a desired grasping position qd and a desired grasping force Fd of a target to be operated; calculating a position difference eq based on an actual touch position qs of the dexterous hand and the desired grasping position qd, and calculating an error force eF based on an actual grasping force Fs and the desired grasping force Fd; determining a basic PID control signal up based on the position difference eq, and determining a correction signal uF based on the error force eF; acquiring visual feature data of the actual touch position qs, calculating a force feedback weight wF of the correction signal uF based on the visual feature data; determining a hybrid control signal uh based on the force feedback weight wF, the correction signal uF and the basic PID control signal up; and controlling the dexterous hand to grasp the target to be operated based on the hybrid control signal uh. The application can improve the control precision of the dexterous hand.

A modular six-axis visual-tactile sensor calibration and testing platform includes: a fixed frame and a helical adjustable pressure module mounted thereon, a support device, a six-axis force sensor located within the support device, a precision rotation positioning platform relative to the fixed frame, a high-precision two-dimensional XY positioning platform mounted thereon, and a connector. The connector has a probe at its end, with the probe tip directly applying force to the contact surface. This invention meets the requirements for visual-tactile sensor testing and experimentation. Depending on the different sensing tasks required for testing the visual-tactile sensor, it integrates detection functional structures adapted to different situations.

The present application belongs to the technical field of robot control and embodied intelligence data acquisition, and particularly relates to a multi-modal data acquisition device. The device comprises a handle, a trigger, a connecting rod assembly, a shell, a visual perception module, a claw finger assembly, a translation guide assembly and a controller. The translation guide assembly and the controller are arranged in the shell. Two groups of connecting rod assemblies are arranged at the front end of the shell. The rear ends of the two groups of connecting rod assemblies are connected with the translation guide assembly, and the front ends are connected with two groups of claw finger assemblies. The handle is arranged at the bottom of the shell, and the trigger is arranged on the handle and connected with the translation guide assembly. A high-density array tactile sensor is attached to the surface of the claw finger assembly for real-time acquisition of clamping contact force information. The visual perception module is arranged on the front side of the shell for acquisition of environmental information. The controller records all multi-modal data in real time. The device is operated by hand, realizes opening and closing movement of the clamping jaw, clamps the operated object, and records and transmits the related data generated in the operation process.

The application discloses an aviation part precision assembly method based on multi-modal fusion perception and action generation, takes a visual-language-action model (VLA) as a core, first collects two-dimensional images, three-dimensional point clouds, force / tactile signal multi-modal information and natural language task instructions of aviation part assembly, then respectively encodes the same to generate visual tokens, state tokens and text tokens, jointly models the multi-modal tokens through the VLA model to obtain a fusion representation, generates a continuous assembly action sequence containing grabbing, moving and aligning based on the representation, finally decodes the action sequence into control instructions to drive an executing mechanism to complete precision assembly. The application realizes end-to-end automatic control from environment perception, task understanding to action generation, improves the automation level, execution precision and task adaptability of aviation assembly, reduces the cost of manual demonstration and debugging, and is suitable for aviation part high-precision automatic assembly scenes.

The application relates to the technical field of visual tactile sensors, and provides a visual tactile sensor based on lensless imaging and a three-dimensional contact force reconstruction method.The visual tactile sensor comprises a reflective film layer, a ring-shaped LED array, a marker point array layer and an image sensor; the reflective film layer and the image sensor are arranged on the two sides of the marker point array layer respectively; the distance between the image sensor and the marker point array layer is less than a distance threshold; the ring-shaped LED array is arranged around the marker point array layer; the light emitted by the ring-shaped LED array is reflected to the marker point array layer via the reflective film layer, and then forms a shadow image of the marker points on the image sensor.Because the lensless imaging is adopted, the distance between the image sensor and the marker point array layer can be reduced to compress the sensor thickness, miniaturization integration is realized, and the contradiction between the lens imaging field of view angle limitation and the mechanical assembly complexity is eliminated.

The application discloses a dense normal distribution force field reconstruction method fusing force and visual tactile information, and comprises the following steps: a neural network model with double-mode input is built based on a U-net framework, wherein the double-mode input comprises a tactile image and a normal resultant force; training data is prepared to train the neural network model into a reconstruction model outputting a dense normal distribution force field; the neural network model is trained by using a training data set to obtain the reconstruction model; the normal force and the tactile image when the robot end contacts an object are acquired in real time, and the trained reconstruction model is used to predict the dense normal distribution force field when the robot end contacts the object. The application can realize real-time dense normal distribution force field reconstruction when the robot end contacts the object, and is applied to contact object shape perception, multi-point contact position perception, rigidity judgment and other robot perception tasks.

The application discloses a visual tactile sensor, a light source design method thereof and related equipment. The visual tactile sensor contains a physical light source, the physical light source includes a plurality of light emitting units of a plurality of logical lighting channels, and the method includes the following steps: obtaining a sensor mechanical model, extracting a feasible layout area and a light transmission model from the sensor mechanical model, determining a global objective function by combining a physical light source layout of the feasible layout area, a driving weight parameter and the light transmission model, and solving the global objective function according to a preset evaluation index to determine a layout coordinate of the physical light source and an optimal driving weight. The evaluation index includes condition number minimization and / or uniformity maximization. The application can optimize the illumination light field and improve the detection accuracy of the visual tactile sensor. The application can be widely applied to the technical field of visual tactile sensors.

This invention discloses a method and system for planning the grasping of unknown objects using visual-tactile multimodal perception. The method includes: acquiring continuous implicit surface representations and semantic-level grasping prior features of the target object; and conducting exploratory contact with the target object based on the semantic-level grasping prior features. A cost function is constructed between visual and tactile nodes based on the distance between visual and tactile nodes after nonlinear projection, and the angular deviation between the visually estimated surface normal and the tactilely measured normal, to obtain a multimodal fusion feature representation. A dynamic availability tensor field is constructed, and the mechanical stability margin is calculated using a pre-set differentiable mechanical verification model to search for the optimal grasping pose. This invention alleviates the scale and representation differences between globally sparse visual data and locally dense tactile data by constructing a visual-tactile cross-modal optimal transmission network, achieving a smooth mapping of local tactile physical features to a global implicit geometric surface.

This invention relates to the field of data security training and assessment technology, and discloses a holographic perception assessment method and system for data security. The holographic perception assessment method for data security includes: constructing a multi-sensory channel security mapping model to map data security risk characteristics to visual, auditory, and tactile perception characteristics; performing personalized perception intensity adjustment and semantic relevance optimization based on user cognitive characteristics; realizing dynamic simulation and interactive experience of known and unknown security threats through security hazard simulation technology; and establishing a unified cross-scenario perception assessment framework. This invention breaks through the limitations of traditional single-sensory channels, realizes multi-dimensional collaborative transmission of security knowledge, enhances users' intuitive understanding and in-depth cognitive ability of data security risks, and provides a systematic solution for improving the overall security awareness of organizations.

Traditional rehabilitation methods lack interest, and the training enthusiasm of the elderly is low; the existing training equipment is large in size and high in cost, and is not suitable for family use; the existing VR training system lacks tactile feedback and cannot realize multi-sensory coordinated stimulation; there is a lack of professional solutions and quantitative evaluation means for finger fine motor actions. The application discloses a hand training device based on touch-visual double synchronization, and relates to the technical field of medical instruments, which comprises a collecting terminal and a remote processing terminal, the remote processing terminal is arranged on a VR glasses, and comprises a data collection layer, which is used for acquiring the rotation angle, pressing force and hand inclination angle uploaded by the collecting terminal; a data processing and synchronization layer, which comprises an action mapping module, which is used for mapping the collected rotation angle into a grabbing / release action in a VR scene, mapping the pressing force into an object weight perception, and mapping the hand inclination angle into a view angle offset; and a VR scene interaction layer, which is used for realizing the action mapped by the action mapping module in the VR scene.