920 results about "Tactile sensor" patented technology

The invention discloses a manipulator grabbing planning system and method based on visual identification, and relates to the technical field of robots, the manipulator grabbing planning system comprises a visual perception module, a coordinate conversion module, a motion planning module, a control execution module, a tail end perception module and a system integration and communication module; the visual perception module realizes multi-view target detection, semantic segmentation and three-dimensional pose estimation through a binocular depth camera; the coordinate conversion module is used for converting a target pose under a camera coordinate system into a world coordinate under a mechanical arm base coordinate system; the motion planning module is responsible for generating a mechanical arm grabbing path, optimizing a strategy and supporting generalization migration of a multi-form mechanical arm; the control execution module drives the six-axis cooperative mechanical arm to complete the grabbing action, and vision-force mixed feedback closed-loop control is achieved. The tail end sensing module monitors the grabbing state in real time through a touch sensor and dynamically adjusts a grabbing strategy; and the system integration and communication module is used for realizing real-time data interaction and cooperative work among the modules of the system.

The invention discloses a multi-modal collaborative decision-making method and device for a humanoid robot in an industrial scene, and relates to the technical field of robot control, and the method comprises the steps: obtaining a multi-frame monitoring image, force touch sensor data and equipment operation sensor data of a visual camera of the humanoid robot in the industrial scene; extracting feature vectors of the multi-frame monitoring image, the force touch sensor data and the equipment operation sensor data as multi-modal feature vectors; taking the multi-modal feature vector as a training sample to train a cross-modal model, so that the cross-modal model learns an association relationship among vision, haptic and equipment data; obtaining dynamic industrial scene data, extracting feature vectors, and inputting the feature vectors into the trained cross-modal model to generate scene description information; and outputting an operation decision instruction according to the scene description information so as to control the humanoid robot. Through cross-modal feature fusion and dynamic knowledge reasoning, the task execution precision and environmental adaptability of the humanoid robot in a complex industrial environment can be improved.

The invention relates to the technical field of robot decision making, and provides a robot multi-modal fusion autonomous decision making method and system based on a large language model.The method comprises the steps that a robot obtains multi-modal environment information through a visual sensor, a touch sensor, an auditory sensor and a laser radar which are carried by the robot; performing preliminary filtering and noise reduction processing on the original sensor data, and synchronously recording all the sensor data by timestamps; performing space-time semantic alignment on the preprocessed multi-modal data, mapping pixel coordinates of a target in a visual target coordinate quantization original image to a robot coordinate system, performing uncertainty evaluation on a multi-modal signal through a dynamic Bayesian network, and taking entropy or variance as an uncertainty quantitative evaluation index. According to the method, the information quality is improved from a data fusion source, accurate and reliable basic support is provided for subsequent decision making, and decision making errors caused by data deviation are greatly reduced.

A control method for a support robot, the method comprising: in response to recognizing that a target object has an intent to require support, controlling a support robot to move toward the target object, and when the support robot moves to a target movement position, controlling a robot arm to move to a target joint position, wherein the target movement position and the target joint position are both determined on the basis of the relative position between the target object and a target sensor; and in the process of the robot arm moving to the target joint position or after the robot arm moves to the target joint position, when it is detected by means of a tactile sensor that an external force is applied to the robot arm, performing compliance control on the support robot. Further provided are a support robot, an electronic device, a computer storage medium, and a computer program.

The invention discloses a robot hierarchical reinforcement learning variable impedance control method based on vision and touch, and belongs to the field of robot control. According to the method, an upper-layer planning system and a lower-layer variable impedance control system are included, the upper-layer planning system fuses information from a visual sensor and a tactile sensor, feature coding is conducted on a visual image through a variational self-coding model, tactile feedback and mechanical arm state information are combined, and an action strategy adapting to the current environment and corresponding impedance parameters are formulated. And the lower-layer variable impedance control system realizes variable impedance control of the robot based on a depth kinematics and dynamics model, and allows the robot to dynamically adjust the motion and contact force according to the sensed information when interacting with the environment, thereby improving the adaptability to the environment change and the task execution efficiency. According to the method, the success rate and execution efficiency of the robot for executing the complex assembly task in the unstructured environment are remarkably improved, and high robustness is shown.

The invention discloses a joint denoising method for robot visual motion prediction, and the method comprises the steps: constructing a unified generative model through fusing an image and a depth map collected by a depth camera, motion data collected by CAN line communication of a Piper mechanical arm, and a tactile image collected by a Gelsight Mini tactile sensor; the method comprises two steps of data acquisition and input coding, and joint denoising and generation: firstly, multi-modal data are coded into low-dimensional potential representation, and then future images, depth maps, tactile data and robot actions are cooperatively predicted through a joint denoising framework based on Transform. A mask self-attention mechanism is innovatively introduced, information interaction between modes is dynamically adjusted, action generation is guided through tactile feedback, and the force control precision is improved. The model adopts a de-noising diffusion probability loss function to jointly optimize multi-modal prediction, so that the output consistency is ensured. According to the method, the robustness and the accuracy of flexible operation of the robot are remarkably improved.

The invention discloses a five-finger bionic dexterous hand with distributed touch perception and a control method. The five-finger bionic dexterous hand comprises a palm part; fingers of the thumb unit, the index finger unit, the middle finger unit, the ring finger unit and the little finger unit respectively comprise a three-stage series flexion and extension mechanism which is sequentially composed of a third knuckle, a second knuckle and a first knuckle from a metacarpophalangeal joint, and each three-stage series flexion and extension mechanism has three flexion and extension freedom degrees and one deflection freedom degree; the tendon rope driving units are arranged in the palm part; the palm center camera is integrated on the front face of the palm part and used for collecting visual information of an operation area; the 15 air pressure type touch sensor modules are installed on the third knuckles, the second knuckles and the first knuckles of all the finger units in a distributed mode, and each air pressure type touch sensor module is integrated with a pressure and temperature double sensing unit and used for sensing contact pressure and environment temperature; the control module is integrated on the back face of the palm part and is in signal connection with the palm center camera, the air pressure type touch sensor modules and the tendon rope driving unit.

The invention discloses an ultrasonic enhanced tactile sensor system, a sensor and a sensing method. The tactile sensor system comprises a multi-mode sensing unit and a processing unit. The multi-mode sensing unit integrates an ultrasonic module and a visual tactile sensing module, ultrasonic acoustic impedance matching and optical imaging transmission are achieved through an acousto-optic matching structure, and ultrasonic echo signals and tactile deformation images can be synchronously collected. In the processing unit, an event triggering module detects a contact event based on a tactile image and generates a switching signal, and a self-adaptive processing module dynamically controls an ultrasonic mode, distance measurement during non-contact, and switching to object internal imaging or material property identification during contact according to the switching signal. And the three-level fusion processing module outputs a multi-modal sensing result through signal-level synchronous acquisition, feature-level cross-modal fusion and decision-level task switching. According to the system, multi-dimensional perception of proximity sense, surface texture and deep information is realized, and the pre-judgment and accurate operation capability of the robot in a complex scene is improved.

The invention relates to a robot dynamic grabbing control method based on visual sense and tactile sense depth fusion. The method comprises the following steps that firstly, based on visual information, surface geometric features of a target object are extracted, grabbing adaptability scores are calculated, an optimal grabbing point is selected, and a collision-free approaching track is planned; secondly, in a contact establishment stage, detecting a contact event through a touch sensor, fusing vision and touch data to unify a coordinate system, calculating a vision-touch consistency score, and applying an initial grabbing force; and thirdly, in the stable holding stage, slip detection is conducted through wavelet packet energy entropy and pressure gradient, and the grabbing force and impedance parameters are dynamically adjusted in combination with self-adaptive impedance control. According to the method, intelligent control over the whole process from approaching to contact to stable holding is achieved, and the adaptability, stability and safety of robot grabbing in the dynamic environment are improved.

The present invention relates to an advanced augmented reality, AR, apparatus featuring a video see-through AR, VST-AR, device that captures images of a real-world environment and displays augmented versions on a screen. The system includes a corporal entity, such as an empty laboratory vessel, equipped with a unique identification marker. This marker provides real-time spatial data for generating visual augmentations representing virtual matter within the vessel. The apparatus integrates various sensors, including an inertial measurement unit, IMU, capacitive tactile sensors, and a thermal diode, to dynamically adjust visual augmentations based on user interactions. Additional sensory enhancements include an olfactory output fan, an eccentric rotating mass, ERM, motor, and auditory outputs. Wireless connectivity via IEEE 802.11 (Wi-Fi) and IEEE 802.15.1 (Bluetooth) ensures seamless communication between components. This AR system is particularly valuable for educational and laboratory settings, offering realistic, interactive simulations.

A visual tactile sensor, a manipulator and a control method thereof, the visual tactile sensor comprising: a contact layer, at least a portion of which is transparent; when the contact layer is subjected to external force, at least one part of a plurality of markers in the marker array generates displacement and / or deformation; and the image acquisition unit is used for performing image acquisition on the marker array and performing image acquisition on the outside through the transparent part of the contact layer. The visual tactile sensor provided by the invention not only can capture the external visual information before operation to identify the operation object, but also can provide the tactile information in the operation process, so that the perception capability of an operation system is enhanced, and the operation success rate is improved.

The invention discloses an intelligent navigation method and system, and relates to the field of intelligent robot control. The method comprises the following steps: acquiring multi-modal data acquired by a touch sensor, a visual sensor and a force sensor; if the matching degree of the tactile data and the visual data is lower than a preset threshold value, adjusting the tactile data by adopting a weighting coefficient to obtain a fusion feature vector; a three-dimensional point cloud map is generated according to the fusion feature vector, and if it is detected that the dynamic change rate of the point cloud map is higher than a preset threshold value, an incremental updating algorithm is triggered to correct the environment model; extracting a terrain feature set from the environment model, and generating a candidate path set by adopting a reinforcement learning algorithm; and if the passing cost value of the candidate path is lower than a preset threshold value, smoothing the candidate path to obtain an optimized navigation path. According to the invention, accurate navigation and path planning in a complex environment are realized, and the adaptability and robustness of a navigation system are improved.

The invention relates to the technical field of artificial intelligence, and discloses a multi-modal man-machine interaction interface and behavior prediction system, which comprises a multi-modal data acquisition module used for acquiring visual, auditory and tactile data of a user in real time; the data synchronization and preprocessing module is used for carrying out time synchronization on the collected multi-modal data and carrying out denoising and error correction on the data; the data fusion module is used for fusing different modal data and extracting core features of the data; the behavior prediction module is used for performing user behavior prediction based on the fused data; and the system optimization module is used for dynamically adjusting the interaction mode and function of the system based on the behavior prediction result so as to realize personalized user experience. By adopting a multi-modal data fusion technology, data from a visual sensor, an auditory sensor and a tactile sensor are efficiently fused, core features are extracted, and a more accurate user behavior prediction effect is achieved.

The invention relates to the technical field of sensors, and provides a visual tactile sensor and a robotic.The visual tactile sensor comprises a shell, an elastic body, a light-transmitting plate, a camera, a light source and a light uniformizing plate, the camera, the light source and the light uniformizing plate are arranged in the shell, the elastic body is connected to the surface of the front end of the light-transmitting plate, and the light-transmitting plate is embedded in the front end of the shell; the elastic body protrudes out of the front end surface of the shell, the light-transmitting plate is made of light-transmitting materials, one end of the light uniformizing plate corresponds to the light source, the other end of the light uniformizing plate corresponds to the side portion of the light-transmitting plate, and the light-transmitting plate is used for transmitting light from the side portion of the light-transmitting plate to the front end surface of the light-transmitting plate. Therefore, more uniform light can be obtained to irradiate the surface of the elastic body connected to the surface of the front end of the light-transmitting plate, the light supplementing uniformity of the light source in the visual tactile sensor is effectively improved, the imaging quality of the elastic body of the visual tactile sensor is further improved, and meanwhile, the precision and the stability of the visual tactile sensor are improved.

An optical tactile sensor comprising a contact part having a contact surface that comes into contact with an object, the contact part being deformed in response to a contact state of the object with respect to the contact surface; a marker disposed on the contact part, and configured to be displaced in response to deformation of the contact part; and an imaging unit configured to capture an image of the marker from a side opposite to a side where the object comes into contact, wherein the contact part allows light in a first wavelength range to pass therethrough, and blocks light in a second wavelength range different from the first wavelength range, the marker reflects light that is incident on the marker from the opposite side, and the imaging unit receives the light reflected by the marker and the light in the first wavelength range, and generates an image.

The invention discloses a multi-mode tactile and visual fusion identification method for medicine box braille salient point characters. The method comprises the following steps: acquiring three-dimensional morphology and pressure data of Braille salient points through a tactile sensor, acquiring an image and the like through a visual sensor, and performing preprocessing (improved U-Net segmentation, illumination correction and the like); hardware is utilized to trigger synchronization, an ICP algorithm is improved to complete space-time alignment, and a feature level (attention mechanism) and a decision level (D-S evidence theory) are fused; character semantics are analyzed by means of a multi-mode neural network, and multi-channel feedback is visualized through voice, tactile vibration and AR. A dynamic optimization mechanism is set, and sensor scheduling, incremental learning and safety verification are covered. The system comprises a tactile / visual perception module, a data processing module (FPGA + NPU), a feedback execution module and a communication module. According to the scheme, the problems of low traditional recognition precision and poor environmental adaptability are solved, high-precision medicine box braille recognition is achieved, and a reliable scheme suitable for multiple scenes is provided for safe medicine use of visually impaired people.

The invention relates to the technical field of robots. According to the touch sensing fingertip device, protection of a touch sensor in fine operation of a mechanical arm is achieved, and objects of sheet structures can be grabbed more accurately. Specifically, through the bionic fingernail part installed on the back side of the fingertip body, the protection part of the bionic fingernail part extends forwards and exceeds the highest point of the touch sensor, and a cantilever protection structure similar to a human fingernail is formed; the protection parts form a physical barrier (for example, the protection parts are preferentially contacted during grabbing) in a non-contact state by utilizing a reasonable coverage range of the bionic nail part, and meanwhile, when the protection parts are matched with fingers on the opposite side, the two corresponding protection parts can be matched to grab an object of a sheet-shaped structure; according to the design, the durability of the sensor in a complex environment is improved, and the accuracy of grabbing the sheet-shaped objects is improved.

The invention discloses a shoe tree grabbing robot track planning system and method based on multi-modal sensing. The system comprises a shoe tree storage area, a mechanical arm grabbing assembly, a multi-modal sensing device, a posture recognition module, an interference sensing module, a grabbing point optimization module, a stable window generation module, a path planning module, a fine adjustment control module and a self-learning module. The system extracts the posture of the shoe tree by fusing visual images, structural sound waves and infrared contour information, then utilizes interference fringes to analyze and judge the material and friction characteristics, combines stability to judge and screen a grabbable target, and executes posture fine adjustment and fitting at the tail end of a clamping jaw through a touch sensing film. And the grabbing failure information is used for updating the strategy database to realize self-adaptive optimization of the grabbing strategy. The method is suitable for grabbing special-shaped objects and is high in robustness and success rate.

A tactile sensor, consisting of a flexible main body with embedded magnets, soft magnetized cilia on the main body, and magnetic sensing elements embedded on the substrate layer. The magnetic sensing elements produce signals in response to the changes in the magnetic field. Tactile interaction with the main body and cilia displaces the magnetic material and changes the magnetic field. Cilia are sensitive to sub-mN force magnitudes, the main body is sensitive to larger range of force magnitudes. Multiple sensing elements, magnets and cilia may be distributed over large and curved surfaces. Three-axial forces can be measured using the sensor.

A topographical measurement system may include a tactile sensor using a contained fluid as an imaging medium.

The invention discloses an unmanned aerial vehicle high-precision spacer dismounting and mounting control method based on tactile feedback, and belongs to the technical field of intelligent operation and maintenance of high-voltage power transmission and distribution lines. According to the method, a multi-mode tactile sensor array and a self-adaptive force control mechanical arm are carried through an unmanned aerial vehicle, three-dimensional force, torque and surface deformation data when the mechanical arm makes contact with a spacer are collected in real time, the action of the mechanical arm is adjusted in combination with a dynamic impedance control algorithm, and high-precision disassembly and assembly operation is achieved. According to the method, the problems of large positioning error and poor operation stability caused by lack of real-time force feedback of a traditional unmanned aerial vehicle disassembling and assembling method are solved, and the operation precision and safety in a complex environment are remarkably improved.

The invention discloses a robot system with tactile perception, and belongs to the technical field of tactile robots. The problem that an existing robot is poor in environment pressure sensing capacity is solved. Comprising a head module, an integrated vision-inertial navigation sensing module and a head controller, two side surfaces are provided with a pair of rotatable connecting ends, and the outer surface is provided with a tactile sensor; the head controller transmits data to the upper computer through the communication module; the driving body module comprises a plurality of modularized joints which are connected end to end, touch sensors are arranged on the two side faces of each modularized joint, and the adjacent modularized joints are orthogonally connected; the modularized joint is integrated with a joint controller and a joint steering engine; the tail module integrates a tail controller and a tail steering engine, and tactile sensors are arranged on the two side faces of the tail module. The tail controller carries out data transmission with the head controller through the adjacent previous joint controller in sequence, and the head controller receives a control instruction of the upper computer and carries out data transmission with all other controllers. According to the invention, the environmental pressure can be sensed in all directions.

The invention relates to the technical field of robot tactile sensing, in particular to a bionic fingertip tactile sensor based on a fiber bragg grating and a sensibilization packaging method thereof.The bionic fingertip tactile sensor based on the fiber bragg grating comprises a flexible contact layer and a strain layer, and micro-column structures arranged in a rectangular array are arranged in the flexible contact layer; the flexible contact layer is seamlessly combined and covers the upper surface of the strain layer, a fiber bragg grating is fixedly packaged in the strain layer right below the micro-column structure, a tough supporting layer is fixed on the upper surface of the strain layer, a bionic three-layer composite packaging system is adopted, and the flexible contact layer is made of low-elasticity-modulus silica gel with the shore hardness of A10-A80. Stress enhancement and curved surface self-adaptive attachment are achieved, the stress transmission efficiency of the strain layer is improved by more than three times of that of a traditional design through a micro-column array structure, the tough supporting layer provides a rigid protection substrate, overload deformation is effectively restrained, and reliable tactile feedback is provided for precise operation of a robot.

The invention relates to the technical field of sensors, in particular to a bionic hair tactile sensor and application, and the sensor comprises a substrate and a bionic cilia array arranged on the substrate; the bionic cilia array comprises a plurality of flexible bionic cilia arranged in an array, and a marking layer is arranged at the bottom of the flexible bionic cilia; an optical system is integrated in the substrate, the optical system comprises an imaging module and a micro prism, and the micro prism reflects the deformation of the marking layer at the bottom of the flexible bionic cilia to the imaging module; and the imaging module is used for capturing deformation data of the marking layer and sending the deformation data to the deep learning algorithm module to realize decoupling and resolution of three-dimensional force. According to the invention, high-resolution multi-dimensional force detection and direction detection are realized through mark point change caused by cilium deformation, and the array design is expanded to a sliding sensing function and a multi-modal object identification function.

The invention discloses a robot finger tactile perception method and system based on artificial intelligence, and the method comprises the following steps: S1, obtaining visual data and tactile perception initial parameters of a to-be-interacted object of a robot finger, carrying out the preprocessing of the visual data, and extracting the appearance features of the object, performing initial configuration on the touch sensing initial parameters; and S2, object recognition is conducted on the preprocessed visual data through a machine learning model, and the material category, morphological characteristics and surface attributes of an object are determined. According to the robot finger tactile perception method and system based on artificial intelligence, by combining a visual system and tactile sensor data and utilizing a convolutional neural network and Transform fusion architecture to extract local textures and global morphological features of an object, high-precision recognition of material categories, morphological features and surface attributes of the object is realized.

The invention discloses a bionic finger, a hand motion control device, a bionic hand control method and a robot, and belongs to the technical field of bionic robotics.The device comprises a plurality of flexible electric actuators which comprise electric actuating materials capable of actively deforming under excitation of an electric field; one end of the flexible electric actuator serves as a fixed end to be connected to the finger adaptation base, and the other end of the flexible electric actuator serves as a free end to be connected to the knuckle structure through a force transmission path. Wherein each flexible electric actuator can be independently controlled, and the plurality of flexible electric actuators cooperatively drive the knuckle structure to generate bionic motion; the passive constraint structure is configured to provide anisotropic mechanical constraint for active deformation of the flexible electric actuator, so that the active deformation is guided and converted into driving displacement or driving force in the first preset direction; and the guide structure is used for guiding the deformation direction of the flexible electric actuator. The device can integrate a strain sensor, an angle sensor and a touch sensor, and realizes closed-loop cooperative control through a controller.

The invention relates to a flexible dual-mode capacitive touch sensor based on a three-gradient micro-dome porous structure. The sensor adopts a sandwich structure and comprises an upper electrode layer, a TGP dielectric layer and a lower electrode layer, wherein the upper electrode layer and the lower electrode layer respectively cover the upper surface and the lower surface of the TGP dielectric layer; the TGP dielectric layer comprises a gradient part and a porous part, the porous part is located on the upper surface of the lower electrode layer, and the gradient part is distributed on the porous part; the gradient part comprises three kinds of micro-dome structures and flat-top protrusions. Different pressure working characteristics of the two structures are fully utilized, so that the porous structure works in a low-pressure area, the gradient structure works in a medium-high pressure area, the performance of the porous structure and the gradient structure is synergistically enhanced, the linear range of the sensor is widened, and the sensitivity is further improved.

A flexible touch sensor with a micro tenon-and-mortise structure comprises an upper flexible sensing substrate, a lower flexible sensing substrate, an interdigital electrode and a flexible bottom film. The interdigital electrode comprises a single-face conductive finger electrode and a double-face conductive finger electrode, the lower surface of the single-face conductive finger electrode is conductive, and the upper surface of the single-face conductive finger electrode is non-conductive. The flexibility of the upper flexible sensing substrate is higher than that of the lower flexible sensing substrate, the lower surface of the upper flexible sensing substrate and the upper surface of the lower flexible sensing substrate are respectively provided with convex and concave microstructures with the same characteristic size, and the surfaces of the microstructures are respectively provided with a conductive film layer. The upper surface of the interdigital electrode is in contact with the lower surface of the upper flexible sensing substrate, the lower surface of the interdigital electrode is adhered to the upper surface of the lower flexible sensing substrate, and a gap is formed between the upper flexible sensing substrate and the lower flexible sensing substrate due to the thickness of the interdigital electrode. The invention has the advantages of high sensitivity, low detection limit and multi-mode signal response monitoring capability, and belongs to the technical field of flexible sensing.

A robot-based assembly control method includes providing a plurality of neural network models comprising at least a reinforcement learning network model and a tensor fusion network model. Training data for the models includes visual data from the vision device, tactile data from the tactile sensor, motion feedback data from the robot and torque feedback data from the robot. An assembly control system based on the robot, which includes a clamping effector at an end of a movable part of the robot, wherein a tactile sensor and a soft rubber pad are provided on an inside clamping portion of the clamping effector from inside to outside is described. Based on fusing the visual information with the external forces of other dimensions indirectly obtained from changes in tactile signals, representation vectors can be used to generate appropriate instructions of robot motion for flexibly adjusting the insertion force to complete the assembly.

The invention belongs to the technical field of robots, and particularly discloses a multi-mode dexterous hand fusing visual tactile sense and palmtop eye and application thereof.The multi-mode dexterous hand comprises a hand body, a visual tactile sensor, a palmtop eye module and a processor, the visual tactile sensor comprises a skeleton, an elastic body, RGB and an NIR camera, the skeleton is of a finger-shaped structure, and the skeleton and the hand body form a hand structure; the elastomer covers the outer side of the framework, and a diffuse reflection coating covers the surface of the elastomer; the RGB camera and the NIR camera are installed in the framework and used for obtaining an RGB image and an NIR image when the first object makes contact with the elastic body; the handheld eye module is installed in the middle of the hand body, comprises a second RGB camera and a TOF camera and is used for taking a second object RGB image and sparse depth; and the processor determines the first object surface three-dimensional point cloud and the second object depth map to realize pose alignment of the two. According to the invention, touch-vision-depth multi-mode perception can be realized, and the precision and robustness of the manipulator are improved.