136 results about "Touch Perception" patented technology

A visual tactile sensor, a manipulator and a control method thereof, the visual tactile sensor comprising: a contact layer, at least a portion of which is transparent; when the contact layer is subjected to external force, at least one part of a plurality of markers in the marker array generates displacement and / or deformation; and the image acquisition unit is used for performing image acquisition on the marker array and performing image acquisition on the outside through the transparent part of the contact layer. The visual tactile sensor provided by the invention not only can capture the external visual information before operation to identify the operation object, but also can provide the tactile information in the operation process, so that the perception capability of an operation system is enhanced, and the operation success rate is improved.

The invention provides a control method and system of a double-arm clothes folding robot, and relates to the technical field of robot control, and the control method constructs a perceived control closed loop by fusing visual, force and tactile multi-modal data. The touch sense and the force sense make up the blind area of the vision in the self-shielding scene, and grabbing, sliding and pleating perception is improved. Based on clothing joint characterization of a visual semantic network, geometric morphology, semantic components and confidence coefficient are structured and coded, a robust state fusing prior and real-time observation is provided for planning, and decision errors caused by deformation and noise are reduced. Long-time-history folding is decomposed into sub-target sequences through a hierarchical strategy, action closed-loop correction is conducted through multi-modal data, and cooperative control over grabbing, folding and pleating is achieved. And finally, a torque instruction is generated based on a control algorithm, interaction compliance and safety are ensured, and the folding success rate and operation stability of the double-arm clothes folding robot under different materials, sizes and initial forms are integrally improved.

The invention relates to the technical field of image analysis, and discloses an end-to-end visual tactile perception method and system based on a morphology-force field analysis model, a terminal and a storage medium, and the method comprises the steps: employing a morphology reconstruction module to achieve the analysis of a micron-order contact surface, employing a force field analysis model to achieve the precise analysis of a force field in each direction, and taking the two results as a data set, inputting and mapping the image into contact morphology, normal and shear force distribution through an end-to-end network, integrating the high efficiency of data driving and the reliability of physical consistency, and finally introducing partial differential equation residual error sum into a loss function to carry out optimization. And the morphology and the force field meet the set physical consistency on the whole. According to the method, micron-sized morphology analysis and multi-axial force synchronous estimation are realized, the real-time robust performance under a high-frequency dynamic task is realized, the cost of tactile perception is reduced, and meanwhile, the accuracy of a tactile perception result is improved.

The invention discloses a robot finger tactile perception method and system based on artificial intelligence, and the method comprises the following steps: S1, obtaining visual data and tactile perception initial parameters of a to-be-interacted object of a robot finger, carrying out the preprocessing of the visual data, and extracting the appearance features of the object, performing initial configuration on the touch sensing initial parameters; and S2, object recognition is conducted on the preprocessed visual data through a machine learning model, and the material category, morphological characteristics and surface attributes of an object are determined. According to the robot finger tactile perception method and system based on artificial intelligence, by combining a visual system and tactile sensor data and utilizing a convolutional neural network and Transform fusion architecture to extract local textures and global morphological features of an object, high-precision recognition of material categories, morphological features and surface attributes of the object is realized.

The invention provides an isomorphic sensor dexterous hand-tactile glove system and a data acquisition method, and belongs to the technical field of tactile sensing, the system comprises a tactile data acquisition module which comprises a glove main body and a fixed flange arranged on the glove main body, a tactile sensor array is arranged on the palm side of the fixed flange and used for collecting tactile data of interaction between the glove body and an object; the visual perception module comprises an infrared camera array and an RGB (Red, Green and Blue) camera array, and is used for tracking marking balls arranged on the fixed flanges so as to solve pose data of the fixed flanges and collecting RGB images of interaction between the glove main body and an object; and the multi-modal data processing module is used for receiving the tactile data, the pose data and the RGB image, realizing space-time alignment of the data and generating a standardized data set. The system is simple in structure, rapid in deployment and high in acquisition safety, and can be suitable for tactile big data acquisition under multi-scene and multi-task conditions.

The invention discloses a cognitive impairment risk intelligent matching intervention system based on multi-modal data, and belongs to the technical field of cognitive impairment, and the cognitive impairment risk intelligent matching intervention system specifically comprises the following steps: collecting a tactile interaction sequence and a face video data stream when a user executes a cognitive intervention task, and constructing an original multi-modal data set; extracting tactile operation track features and reconstructing a heart rate variability sequence, and generating a synchronous multi-mode feature set; identifying a difference interval of state fluctuation and extracting behavior and physiological features in the interval to form an instant physical and mental load parameter set; in combination with an interaction efficiency index in the task execution record, generating a joint state feature vector, and outputting a personalized adaptation instruction through a pre-trained sub-state-parameter association model; and constructing an adaptive strategy model by using the multi-modal features and an adaptive instruction to realize millisecond-level dynamic adjustment from the real-time multi-modal features to task parameters. According to the method, real-time perception and accurate matching of the cognitive emotion load of the user are realized, and the individuation degree and instantaneity of intervention are improved.

The invention relates to a visual sense and tactile sense fusion object identification method and system based on wavelet enhancement Transform. The method comprises the following steps: acquiring an original visual image and a tactile sense time sequence of an object to be identified; generating a visual token sequence and reconstructing visual features through visual wavelet enhancement processing based on the original visual image, and generating a tactile token sequence and reconstructing a tactile time sequence through tactile wavelet enhancement processing based on the tactile time sequence; based on the visual token sequence and the reconstructed visual features, generating a two-dimensional visual feature map by using a visual Transform encoder; based on the tactile token sequence and the reconstructed tactile time sequence, generating a tactile feature map by using a tactile Transform encoder; semantic alignment and fusion are carried out based on the two-dimensional visual feature map and the tactile feature map to obtain fusion features, and object recognition is realized based on the fusion features; the system is used for implementing the method. Compared with the prior art, the method improves the recognition precision.

The application belongs to the technical field of two-dimensional material devices, and discloses a strain-enhanced multi-sensory fusion optoelectronic synapse device and a preparation method and application thereof. The optoelectronic synapse device comprises, from bottom to top, an insulating layer, an electrode and a functional layer. The material of the functional layer is an in-plane polarization two-dimensional ferroelectric material, and the functional layer is applied with a tensile strain along the direction of its own polar axis. The optoelectronic synapse device simulates the perception of visual information through the photoconductivity effect of the functional layer and simulates the sense of touch through the tensile strain. The synergistic effect of the photoconductivity effect and the tensile strain significantly enhances the persistent photoconductivity (PPC) of the synapse device, and the enhancement processing of a fingerprint image is realized through the characteristics of the device. At the same time, by applying the tensile strain along a specific direction, the in-plane polarization intensity is significantly improved, the constraint of the depolarization field is avoided, the stability and performance of the ferroelectric device are improved, and a new way for improving the performance of the optoelectronic synapse device is provided.

The utility model relates to the technical field of sensors, in particular to a visual touch sensor. According to the utility model, the colloid on the transparent plate is in contact with an object, the single-sided mirror is arranged opposite to the colloid on the opening, the camera component performs visual detection on the deformation condition of the colloid reflected by the single-sided mirror, and the effect of clamping different objects is achieved through the deformation of the colloid, so that the tactile perception capability of the visual tactile sensor is realized; the robot is simple in structure, low in cost and high in adaptability, and can be widely applied to the fields of object grabbing, flexible operation, touch perception in rehabilitation prostheses and the like; and meanwhile, the number of parts required by the visual tactile sensor is reduced, the working stability of the visual tactile sensor can be effectively improved, the failure rate is low, and the reliability is relatively high.

The invention provides an object sensing method and system based on proximity vision and tactile deformation, which are applied to the technical field of tactile sensing, and the method comprises the steps: obtaining a binocular image collected by a visual tactile sensor after a target object is in contact with a preset visual tactile sensor; performing visual feature extraction based on the binocular image to obtain a binocular feature point two-dimensional coordinate of the target object; extracting a tactile mark based on the binocular image to obtain a two-dimensional coordinate of a tactile mark point of the target object; through a three-dimensional reconstruction model based on ray tracing, based on the two-dimensional coordinates of the binocular feature points, determining approximate three-dimensional visual perception of the target object in the non-contact area; and determining contact three-dimensional deformation perception of the target object in the contact area through a three-dimensional reconstruction model based on ray tracing and based on the two-dimensional coordinates of the binocular feature points. According to the invention, the perception capability of the visual tactile sensor on the object state and the interaction degree can be improved.

The invention particularly relates to an intelligent garbage classifying and sorting system based on machine vision, and relates to the technical field of intelligent garbage classification and industrial automation. A digital twin construction module; an adaptive control and optimization module; and a predictive maintenance and health management module. According to the method, vision-touch-acoustics multi-mode perception is deeply fused, so that the identification limitation of a traditional single vision sorting system is solved; after the visual identification and the tactile feature are subjected to weighted fusion, the material classification accuracy is improved, and the problems of misjudgment and missed judgment under the scenes of oil contamination, light reflection, form deformation and the like are effectively avoided.

The invention discloses a visual tactile sensor fused with an optical waveguide and a preparation method thereof, and belongs to the technical field of intelligent tactile sensing. The visual tactile sensor comprises a fixed shell, and the top end of the fixed shell is open; the composite tactile sensing layer is arranged at the opening position of the fixed shell, and the composite tactile sensing layer is sequentially composed of a shading layer, a reflecting layer, a red-green spaced fluorescent layer, an elastic deformation layer and a supporting layer from top to bottom; and the optical lighting and sensing component comprises a blue LED light bar, an array light intensity sensor and a camera, the blue LED light bar and the array light intensity sensor are arranged on the two corresponding sides of the composite touch sensing layer respectively, and the camera is arranged at the bottom in the fixed shell. Through fusion of rapid and stable light intensity detection characteristics of optical waveguide sensing and pressure distribution detection capability of the visual tactile sensor, high-resolution space and texture identification and high-sensitivity mechanical response detection are simultaneously realized in a single sensor architecture.

The invention discloses a visual tactile sensor, an electronic skin and a robot, relates to the field of sensors, and is used for improving the comprehensiveness and sensitivity of stress detection. The sensor comprises a magnetic film covering the top surface and the side surface, and the magnetic film is prepared by mixing magnetic powder and silicone rubber; the support supports the magnetic thin film, a magnetic display card and a light source are arranged in the support, the visual touch collector collects magnetic induction data and development data of stress of the magnetic thin film, and then the multi-mode pressure detection model analyzes the stress position and / or the stress size. The stress detection sensitivity is high, comprehensive detection of the top face and the side faces can be achieved, and the detection precision is high.

The invention provides an industrial robot self-adaptive grabbing method based on multi-modal sensing, which comprises the following steps of: 1, acquiring multi-modal data of an object and an environment through a multi-modal sensing module; the multi-mode sensing module comprises at least two sensing modules of a visual sensing unit, a touch sensing unit, a force sensing unit and an auditory sensing unit; 2, preprocessing and feature extraction are carried out on the multi-modal data, feature information of different modals is fused through a multi-modal fusion algorithm, and comprehensive object description information is generated; 3, generating an optimal grabbing strategy based on the comprehensive object description information; wherein the grabbing strategy comprises the position and posture of a grabbing point, a grabbing path and grabbing force; and fourthly, the industrial robot executes the optimal grabbing strategy, the first step to the third step are repeated, and the grabbing strategy is adjusted in a self-adaptive mode till grabbing is completed. Stable and efficient grabbing can be achieved.

An embodiment of the utility model discloses a visual tactile sensor, which relates to the technical field of sensors and comprises a base, a flexible plate, an image acquisition device and a plane mirror. The flexible plate is arranged on the base, the image acquisition device is arranged on the base, the image acquisition device is located on one side of the flexible plate, and the plane mirror is obliquely arranged on the upper side of the flexible plate. The layout between the flexible plate and the image acquisition device can be changed through the plane mirror, the size of the visual tactile sensor in the height direction is greatly reduced, and the size of the visual tactile sensor is reduced. According to the scheme, through the innovative structural design of reflection imaging of the plane mirror, on the premise that the tactile perception precision is ensured, the problem that the size of a traditional visual tactile sensor is too large due to vertical imaging layout is solved, meanwhile, multiple technical requirements of environmental adaptability, cost controllability and the like are taken into consideration, and the visual tactile sensor is suitable for popularization and application. And better hardware support is provided for scenes such as refined operation and man-machine interaction of the robot.

The invention discloses a vision-based high-density three-dimensional force sensing array system and a three-dimensional force distribution identification method thereof. The system comprises a first force sensing layer, a second force sensing layer, a support force bearing layer, an LED lighting unit, a light-transmitting reference plate, a camera and a supporting shell box body. The second force sensing layer is a high-density array formed by independently distributed cylinder and hemisphere elastic structure units; and the light-transmitting reference plate, the LED lighting unit and the camera form an optical imaging assembly. When an external object is in contact with the surface of the array, each structural unit deforms independently, a camera captures a deformation image, features are extracted through an image processing algorithm, and three-dimensional force distribution of a contact area is solved according to a pre-calibrated image-mechanics mapping model. The system can reconstruct the three-dimensional contour form of a contact object in real time and perceive the force distribution characteristics of the contact object, and a high-precision and low-cost force touch perception solution is provided for the fields of robot touch recognition, human-computer interaction, safety monitoring and the like.

The invention discloses a small-size visual tactile sensor based on a UV mark, and relates to the field of sensors. The visual tactile sensor solves the problem that the existing visual tactile sensor cannot realize high-precision force and texture perception on the same sensor at the same time, independently or in a time-sharing manner. A UV fluorescence labeling layer which is transparent under visible light and develops under ultraviolet light is arranged on the contact surface of the elastomer; the composite light source module comprises a UV light source unit and an RGB light source unit which can be independently controlled; the UV light source or the RGB light source is turned on through time-sharing switching, so that the image acquisition module can acquire mark point displacement images used for calculating contact force distribution or photometric stereo images used for reconstructing contact surface textures in a time-sharing manner. According to the invention, the problem that a traditional sensor cannot realize lossless switching of force and texture sensing functions on a single device is fundamentally solved by using the invisibility characteristic of the UV mark and a dual-light-source time-sharing working mode. The method is used for robot touch sensing.

The invention discloses a tactile image domain migration method and device based on a multi-scale generative adversarial network, and the method comprises the steps: constructing the multi-scale generative adversarial network, and achieving the domain migration between a simulation tactile image and a real tactile image. The generator takes U-Net as a trunk, introduces a multi-scale stacking module, a multi-stage attention gate mechanism and a channel-space attention module, and improves the reconstruction capability of simulation image textures, illumination and contact areas. And the discriminator adopts a multi-scale discrimination structure to realize the discrimination of image authenticity and detail consistency. Through joint training of joint adversarial loss, loop consistency loss, contact area consistency loss, illumination balance loss and frequency domain loss functions, it is ensured that an output image is consistent with a real image in visual and semantic levels. The method can be widely applied to a robot vision-touch fusion perception task, and the migration performance and robustness of the perception model are remarkably improved under the condition of non-paired data.

The invention discloses a visual touch fusion object material identification method and device based on channel switching, and the method comprises the steps: obtaining a paired data sample which comprises an object and is composed of a visual image and a touch image, carrying out the preprocessing of the paired data sample, and constructing a training data set; constructing a visual touch fusion object material identification network, wherein the visual touch fusion object material identification network comprises a visual feature extraction sub-network, a touch feature extraction sub-network, a channel switching gating fusion module, a space attention module, a feature pyramid fusion module and a classification module; training the visual touch fusion object material identification network by using the training data set; and deploying the trained visual touch fusion object material identification network to a device end, and performing object material identification by using the trained visual touch fusion object material identification network to obtain an identification result. According to the invention, on the basis of the lightweight design of the model, the substance identification accuracy is obviously improved.

The invention provides a multi-mode visual touch fusion sensor of a non-coaxial light path, which realizes multi-mode cooperation of a visual mode and a visual touch mode by performing non-coaxial arrangement on the optical axis of a camera and the central normal of a touch plane on a structural level; a virtual binocular system is constructed under the condition of a single camera by means of a free-form surface light splitting biprism, the number of devices and the assembling freedom degree are reduced, and therefore the integration degree and calibration consistency are improved; a semi-transparent and semi-reflective optical element is matched with an LED light-emitting unit time sequence and a control circuit to realize automatic and rapid mode switching, so that the long-term reliability is improved, and the switching service life is prolonged; through the collaborative design of the self-assembled metal nano reflective coating and the thin polyurethane packaging layer, a high-contrast and wear-resistant deformation image is obtained on the premise of not sacrificing the tactile sensitivity, and finally continuous acquisition of a target pose and contact parameters is realized. And the comprehensive requirements of accurate grabbing and touch closed-loop control on stable imaging, rapid switching, easy integration and high reliability are met.

The utility model discloses a visual tactile sensor, which relates to the technical field of visual tactile sensors and comprises a mechanical component and an optical component. The mechanical part comprises a transparent elastic body and a transparent supporting body for supporting the transparent elastic body; the transparent elastomer is provided with a mark point pattern and a reflection pattern; the optical component comprises a light source, an image acquisition device and a light filtering device; the image acquisition device comprises a first image acquisition unit and a second image acquisition unit; the first image acquisition unit is configured to directly acquire a picture of the transparent elastomer; and the second image acquisition unit is configured to acquire a picture of the transparent elastomer passing through the light filtering device. According to the utility model, lossless texture information and accurate multi-dimensional force information can be synchronously obtained in real time at the same time.

The application discloses a robot complex scene perception system and method based on environment adaptive multi-modal fusion, relates to the technical field of robot perception, and comprises four modules of a flexible sensing matrix, environment feature self-calibration, cross-modal causal reasoning and scene prediction.The flexible sensing matrix collects environment parameters and multi-modal data of vision, inertia and touch.The self-calibration module corrects data errors through environment-error mapping and gradient descent algorithm.The causal reasoning module realizes data fusion and failure completion based on a Bayesian network.The scene prediction module outputs a prediction result through a pre-trained LSTM model and feeds back an optimized collection strategy.The application breaks through the bottleneck of traditional perception technology, improves the environment adaptability, system fault tolerance and decision initiative of complex unstructured scenes, is suitable for scenes such as disaster rescue and outdoor inspection, and helps robots to work autonomously and safely.

This invention discloses a robot visual-tactile perception method and system based on transformer and contrastive learning, belonging to the field of robotics technology. The method introduces an enhanced Swing Transformer model as the perception model. This enhanced model uses the Swing Transformer as the backbone network and incorporates a coordinate attention mechanism. Global information is extracted through the sliding window attention mechanism of the backbone network, while local feature extraction is enhanced through the coordinate attention mechanism. This effectively extracts and fuses global and local features from visual and tactile data. In the pre-training stage, the enhanced Swing Transformer model undergoes multimodal contrastive learning training, specifically intra-modal and inter-modal contrastive learning, generating richer and more accurate discriminative features, thereby effectively improving the robot's intelligent perception and manipulation capabilities. This method can effectively extract features in multimodal data fusion and utilize the correlation and synergistic effects of features between different modalities to achieve deeper multimodal data fusion.

An integrated real-time personal safety system, described herein, is a holistic solution designed to enhance user safety through proactive hazard detection and alerts. This system comprises a data collection unit that features a sensor-embedded wearable device and a set of environmental and physiological sensors, providing comprehensive coverage of the user's surroundings and health status. Aided by wireless communication modules, the system ensures seamless transmission of real-time data to cloud-based services for analysis. The system's processing unit leverages this data to predict potential threats and initiates the signal alert unit to promptly notify the user through visual, auditory, or haptic cues. By bridging the gap between data collection, analysis, and user alert, the system ensures a proactive response to potential hazards.

The invention relates to an enhanced multi-mode robot visual perception system, and aims to remarkably improve the adaptability and perception capability of a robot to a complex environment by integrating various sensors and an advanced data fusion algorithm. The system comprises a high-resolution camera, a multi-line laser radar, an infrared sensor, a sonar, a touch sensor and the like so as to realize comprehensive perception of the environment. An innovative data fusion algorithm can process and integrate data from different sensors in real time, and a consistent and accurate environment model is provided. The system is also integrated with a machine learning module, so that the robot can continuously learn and adapt to new environment characteristics and changes through a deep learning algorithm. A real-time feedback and control mechanism ensures that the robot can quickly adjust behaviors and path planning according to environment changes. Through the design of a user interaction interface, the system state is monitored in real time, and intervention is carried out when necessary. The system provided by the invention adopts a modular design, so that the sensor configuration can be quickly adjusted and upgraded according to different application scenes.

The invention relates to the technical field of robot grabbing and sensing, in particular to a dexterous hand / clamping jaw integrated module and a method for grabbing an object through the dexterous hand / clamping jaw integrated module. An existing robot dexterous hand / clamping jaw depends on external vision and touch perception, a system is complex, a view blind area exists, adaptability is poor, and cost is high. The integrated module is arranged at the fingertip of the dexterous hand or the claw tip of the clamping claw and comprises an elastic body, a cover plate and a module base which are sequentially attached to one another from front to back, the elastic body is provided with a through hole, a forward depth camera module is arranged on the front face of the module base, and the elastic body is provided with a through hole. A window hole matched with the depth camera module is formed in the cover plate, when an object is grabbed, the front face, back to the cover plate, of the elastic body makes contact with the object, and the forward depth camera module forwards obtains image and depth information through the window hole and the through hole. The integrated module simultaneously has visual positioning and force perception, does not need to depend on an external sensor, and is simplified in system, good in visual field and low in cost.

The present invention relates to a liquid detection indicator, configured to provide a visual and / or tangible indication if liquid such as water leaks into a cavity / space. The liquid soluble element can be made to withstand oil, but dissolve completely or partly in water, thereby allowing the liquid detection indicator to be installed either in a dry void, or in a void filled completely or partly with oil such as lubrication / gear oil, this so that if water enters said dry void, or if an amount of water ingress to said oil filled void exceeds a certain value, the liquid soluble element will dissolve partly or complete to release a mechanism allowing for a visual and / or tangible signal, e.g. release plunger.

The present application relates to the technical field of tactile sensing and traditional Chinese medicine intelligent diagnosis, and in particular to a massage method measuring device and recognition method based on a visual tactile sensor, which synchronously collects six-dimensional force / torque data and force application position information during the massage process through the visual tactile sensor, and after data preprocessing and feature extraction, the force application position and massage method are accurately recognized by combining deep learning and random forest algorithm. Experimental verification shows that the force estimation error of the device is about 0.27 N, the torque estimation error is 4.3 Nmm, the force application position recognition accuracy is 97%, and the recognition accuracy of seven traditional Chinese massage methods is 97.3%, solving the problems of traditional Chinese massage method quantization difficulty and low recognition accuracy, and realizing the standardization of massage methods, providing technical support for the standardized teaching, clinical application and intelligent rehabilitation equipment development of traditional Chinese massage.

This invention discloses a guide device and system that converts image signals into vibration signals, relating to the field of assistive technology for the blind. Its key technical points are: the device and system of this invention convert environmental semantic information into perceptible tactile signals, assisting blind people to more comprehensively understand the location of obstacles, road surface properties, and traffic conditions, while simultaneously avoiding dangers on the road. This invention solves the technical problems of low safety and low user acceptance of existing invasive guide devices, as well as poor real-time information transmission in existing guide devices. Furthermore, the printed circuit board assembly of this invention can be flexibly embedded in portable miniature devices, providing blind users with a more natural and convenient guide device.