179 results about "Touch Perception" patented technology

The invention discloses a manipulator grabbing planning system and method based on visual identification, and relates to the technical field of robots, the manipulator grabbing planning system comprises a visual perception module, a coordinate conversion module, a motion planning module, a control execution module, a tail end perception module and a system integration and communication module; the visual perception module realizes multi-view target detection, semantic segmentation and three-dimensional pose estimation through a binocular depth camera; the coordinate conversion module is used for converting a target pose under a camera coordinate system into a world coordinate under a mechanical arm base coordinate system; the motion planning module is responsible for generating a mechanical arm grabbing path, optimizing a strategy and supporting generalization migration of a multi-form mechanical arm; the control execution module drives the six-axis cooperative mechanical arm to complete the grabbing action, and vision-force mixed feedback closed-loop control is achieved. The tail end sensing module monitors the grabbing state in real time through a touch sensor and dynamically adjusts a grabbing strategy; and the system integration and communication module is used for realizing real-time data interaction and cooperative work among the modules of the system.

The invention provides a dexterous hand multi-object stable grabbing method and system based on vision and touch perception. The dexterous hand multi-object stable grabbing method comprises the steps that RGB images and depth maps of target objects are collected through a monocular RGB-D camera, and dense point cloud data are generated; geometric features are extracted by using a three-dimensional point cloud neural network, and composite object distribution features are analyzed in combination with visual information; an initial grasping strategy is generated based on the features, actions are executed, and the joint state and contact force data are synchronously obtained in real time through body sensing and a fingertip force sensor; in the carrying process, multi-dimensional force perception and the joint state are fused to analyze object stress distribution and center-of-gravity shift; vision and force sensing data are used as input. Vision, body sensing and force sensing data are fused, the problems of three-dimensional structure recognition and dynamic stress regulation and control in multi-object composite grabbing are solved, compared with a single sensing scheme, the grabbing planning precision and the self-adaptive capacity in a complex scene are remarkably improved, and the method is particularly suitable for stable carrying of combinations of loose objects such as dinner plates carrying tableware.

The invention discloses a digital twin interaction control system based on robot vision and tactile fusion, and the system comprises a data collection module which is used for collecting visual image data and tactile sensing data; the preprocessing module is used for preprocessing; the multi-mode perception modeling module is used for executing spatial perception modeling and tactile state modeling; the fusion state vector construction module is used for generating a fusion state vector; the digital twinborn mapping module is used for constructing a digital twinborn body; the simulation interaction control module is used for executing interactive action control and generating a control instruction parameter group; the action execution and feedback module is used for controlling the robot to execute the actual action and updating the fusion state vector; and the dynamic evaluation and self-adaptive adjustment module is used for dynamically optimizing and adjusting the fusion state vector. Visual and touch information of the robot is fused, a digital twinning synchronous control mechanism is constructed, and the interactive optimization method which is fast in operation response, high in execution precision and stable in control process is achieved.

A visual tactile sensor, a manipulator and a control method thereof, the visual tactile sensor comprising: a contact layer, at least a portion of which is transparent; when the contact layer is subjected to external force, at least one part of a plurality of markers in the marker array generates displacement and / or deformation; and the image acquisition unit is used for performing image acquisition on the marker array and performing image acquisition on the outside through the transparent part of the contact layer. The visual tactile sensor provided by the invention not only can capture the external visual information before operation to identify the operation object, but also can provide the tactile information in the operation process, so that the perception capability of an operation system is enhanced, and the operation success rate is improved.

The invention provides a touch and visual perception integrated micro-nano robot magnetic field control system and method, and the system comprises a camera which collects the image of a micro-nano robot in a pipeline in real time; the magnetic field generating device generates a gradient magnetic field for driving the micro-nano robot to move through a permanent magnet; the tail end of the mechanical arm is connected with the permanent magnet and adjusts the three-dimensional pose to change the magnetic field distribution; the force feedback equipment is used for providing a touch signal for the micro-nano robot to be in contact with the pipeline boundary; and the computer is configured to identify the aggregation state of the micro-nano robot according to the camera image, increase the resistance of the force feedback equipment and reduce the rotation speed of the permanent magnet when the dispersion state is detected, and reduce the resistance and recover the conventional magnetic field control after the aggregation state is recovered. According to the system, accurate control and abnormal state self-adaptive response of the micro-nano robot in a complex experiment scene are realized through touch-vision collaborative perception and dynamic magnetic field regulation and control.

The invention discloses a wearable path guiding system based on visual-tactile cross-modal mapping. The wearable path guiding system adopts a three-module architecture, namely a visual environment sensing module, a simplified path planning module and a tactile guiding module. The environment perception module constructs an incremental scene map with semantic annotations based on a depth camera. And the simplified path planning module is used for generating a steering vector containing gait information and a steering angle according to the advancing direction and simplified path pointing and performing real-time iterative optimization to realize real-time matching of the advancing direction of the user and the target direction. And the tactile guidance module is used for realizing synchronization of gait and tactile stimulation, generating priority gradient intensity field tactile feedback with a direction indication characteristic according to path deviation information, correcting the advancing direction, and starting the voice auxiliary module when multiple times of tactile correction do not reach the expectation. The method is suitable for a visual signal limited scene, the visual signal is mapped into the tactile signal to realize self-adaptive synchronization with the motion rhythm, and path guidance is completed.

The invention discloses a multi-mode tactile and visual fusion identification method for medicine box braille salient point characters. The method comprises the following steps: acquiring three-dimensional morphology and pressure data of Braille salient points through a tactile sensor, acquiring an image and the like through a visual sensor, and performing preprocessing (improved U-Net segmentation, illumination correction and the like); hardware is utilized to trigger synchronization, an ICP algorithm is improved to complete space-time alignment, and a feature level (attention mechanism) and a decision level (D-S evidence theory) are fused; character semantics are analyzed by means of a multi-mode neural network, and multi-channel feedback is visualized through voice, tactile vibration and AR. A dynamic optimization mechanism is set, and sensor scheduling, incremental learning and safety verification are covered. The system comprises a tactile / visual perception module, a data processing module (FPGA + NPU), a feedback execution module and a communication module. According to the scheme, the problems of low traditional recognition precision and poor environmental adaptability are solved, high-precision medicine box braille recognition is achieved, and a reliable scheme suitable for multiple scenes is provided for safe medicine use of visually impaired people.

The invention provides a control method and system of a double-arm clothes folding robot, and relates to the technical field of robot control, and the control method constructs a perceived control closed loop by fusing visual, force and tactile multi-modal data. The touch sense and the force sense make up the blind area of the vision in the self-shielding scene, and grabbing, sliding and pleating perception is improved. Based on clothing joint characterization of a visual semantic network, geometric morphology, semantic components and confidence coefficient are structured and coded, a robust state fusing prior and real-time observation is provided for planning, and decision errors caused by deformation and noise are reduced. Long-time-history folding is decomposed into sub-target sequences through a hierarchical strategy, action closed-loop correction is conducted through multi-modal data, and cooperative control over grabbing, folding and pleating is achieved. And finally, a torque instruction is generated based on a control algorithm, interaction compliance and safety are ensured, and the folding success rate and operation stability of the double-arm clothes folding robot under different materials, sizes and initial forms are integrally improved.

The invention relates to the technical field of image analysis, and discloses an end-to-end visual tactile perception method and system based on a morphology-force field analysis model, a terminal and a storage medium, and the method comprises the steps: employing a morphology reconstruction module to achieve the analysis of a micron-order contact surface, employing a force field analysis model to achieve the precise analysis of a force field in each direction, and taking the two results as a data set, inputting and mapping the image into contact morphology, normal and shear force distribution through an end-to-end network, integrating the high efficiency of data driving and the reliability of physical consistency, and finally introducing partial differential equation residual error sum into a loss function to carry out optimization. And the morphology and the force field meet the set physical consistency on the whole. According to the method, micron-sized morphology analysis and multi-axial force synchronous estimation are realized, the real-time robust performance under a high-frequency dynamic task is realized, the cost of tactile perception is reduced, and meanwhile, the accuracy of a tactile perception result is improved.

The invention discloses a robot finger tactile perception method and system based on artificial intelligence, and the method comprises the following steps: S1, obtaining visual data and tactile perception initial parameters of a to-be-interacted object of a robot finger, carrying out the preprocessing of the visual data, and extracting the appearance features of the object, performing initial configuration on the touch sensing initial parameters; and S2, object recognition is conducted on the preprocessed visual data through a machine learning model, and the material category, morphological characteristics and surface attributes of an object are determined. According to the robot finger tactile perception method and system based on artificial intelligence, by combining a visual system and tactile sensor data and utilizing a convolutional neural network and Transform fusion architecture to extract local textures and global morphological features of an object, high-precision recognition of material categories, morphological features and surface attributes of the object is realized.

The invention provides an isomorphic sensor dexterous hand-tactile glove system and a data acquisition method, and belongs to the technical field of tactile sensing, the system comprises a tactile data acquisition module which comprises a glove main body and a fixed flange arranged on the glove main body, a tactile sensor array is arranged on the palm side of the fixed flange and used for collecting tactile data of interaction between the glove body and an object; the visual perception module comprises an infrared camera array and an RGB (Red, Green and Blue) camera array, and is used for tracking marking balls arranged on the fixed flanges so as to solve pose data of the fixed flanges and collecting RGB images of interaction between the glove main body and an object; and the multi-modal data processing module is used for receiving the tactile data, the pose data and the RGB image, realizing space-time alignment of the data and generating a standardized data set. The system is simple in structure, rapid in deployment and high in acquisition safety, and can be suitable for tactile big data acquisition under multi-scene and multi-task conditions.

The invention discloses a cognitive impairment risk intelligent matching intervention system based on multi-modal data, and belongs to the technical field of cognitive impairment, and the cognitive impairment risk intelligent matching intervention system specifically comprises the following steps: collecting a tactile interaction sequence and a face video data stream when a user executes a cognitive intervention task, and constructing an original multi-modal data set; extracting tactile operation track features and reconstructing a heart rate variability sequence, and generating a synchronous multi-mode feature set; identifying a difference interval of state fluctuation and extracting behavior and physiological features in the interval to form an instant physical and mental load parameter set; in combination with an interaction efficiency index in the task execution record, generating a joint state feature vector, and outputting a personalized adaptation instruction through a pre-trained sub-state-parameter association model; and constructing an adaptive strategy model by using the multi-modal features and an adaptive instruction to realize millisecond-level dynamic adjustment from the real-time multi-modal features to task parameters. According to the method, real-time perception and accurate matching of the cognitive emotion load of the user are realized, and the individuation degree and instantaneity of intervention are improved.

The invention relates to a visual sense and tactile sense fusion object identification method and system based on wavelet enhancement Transform. The method comprises the following steps: acquiring an original visual image and a tactile sense time sequence of an object to be identified; generating a visual token sequence and reconstructing visual features through visual wavelet enhancement processing based on the original visual image, and generating a tactile token sequence and reconstructing a tactile time sequence through tactile wavelet enhancement processing based on the tactile time sequence; based on the visual token sequence and the reconstructed visual features, generating a two-dimensional visual feature map by using a visual Transform encoder; based on the tactile token sequence and the reconstructed tactile time sequence, generating a tactile feature map by using a tactile Transform encoder; semantic alignment and fusion are carried out based on the two-dimensional visual feature map and the tactile feature map to obtain fusion features, and object recognition is realized based on the fusion features; the system is used for implementing the method. Compared with the prior art, the method improves the recognition precision.

The application belongs to the technical field of two-dimensional material devices, and discloses a strain-enhanced multi-sensory fusion optoelectronic synapse device and a preparation method and application thereof. The optoelectronic synapse device comprises, from bottom to top, an insulating layer, an electrode and a functional layer. The material of the functional layer is an in-plane polarization two-dimensional ferroelectric material, and the functional layer is applied with a tensile strain along the direction of its own polar axis. The optoelectronic synapse device simulates the perception of visual information through the photoconductivity effect of the functional layer and simulates the sense of touch through the tensile strain. The synergistic effect of the photoconductivity effect and the tensile strain significantly enhances the persistent photoconductivity (PPC) of the synapse device, and the enhancement processing of a fingerprint image is realized through the characteristics of the device. At the same time, by applying the tensile strain along a specific direction, the in-plane polarization intensity is significantly improved, the constraint of the depolarization field is avoided, the stability and performance of the ferroelectric device are improved, and a new way for improving the performance of the optoelectronic synapse device is provided.

The utility model relates to the technical field of sensors, in particular to a visual touch sensor. According to the utility model, the colloid on the transparent plate is in contact with an object, the single-sided mirror is arranged opposite to the colloid on the opening, the camera component performs visual detection on the deformation condition of the colloid reflected by the single-sided mirror, and the effect of clamping different objects is achieved through the deformation of the colloid, so that the tactile perception capability of the visual tactile sensor is realized; the robot is simple in structure, low in cost and high in adaptability, and can be widely applied to the fields of object grabbing, flexible operation, touch perception in rehabilitation prostheses and the like; and meanwhile, the number of parts required by the visual tactile sensor is reduced, the working stability of the visual tactile sensor can be effectively improved, the failure rate is low, and the reliability is relatively high.

The invention provides an object sensing method and system based on proximity vision and tactile deformation, which are applied to the technical field of tactile sensing, and the method comprises the steps: obtaining a binocular image collected by a visual tactile sensor after a target object is in contact with a preset visual tactile sensor; performing visual feature extraction based on the binocular image to obtain a binocular feature point two-dimensional coordinate of the target object; extracting a tactile mark based on the binocular image to obtain a two-dimensional coordinate of a tactile mark point of the target object; through a three-dimensional reconstruction model based on ray tracing, based on the two-dimensional coordinates of the binocular feature points, determining approximate three-dimensional visual perception of the target object in the non-contact area; and determining contact three-dimensional deformation perception of the target object in the contact area through a three-dimensional reconstruction model based on ray tracing and based on the two-dimensional coordinates of the binocular feature points. According to the invention, the perception capability of the visual tactile sensor on the object state and the interaction degree can be improved.

The invention particularly relates to an intelligent garbage classifying and sorting system based on machine vision, and relates to the technical field of intelligent garbage classification and industrial automation. A digital twin construction module; an adaptive control and optimization module; and a predictive maintenance and health management module. According to the method, vision-touch-acoustics multi-mode perception is deeply fused, so that the identification limitation of a traditional single vision sorting system is solved; after the visual identification and the tactile feature are subjected to weighted fusion, the material classification accuracy is improved, and the problems of misjudgment and missed judgment under the scenes of oil contamination, light reflection, form deformation and the like are effectively avoided.

The invention relates to a food sensory evaluation method and device based on an attention mechanism, equipment and a medium. The method comprises the following steps: acquiring sensory information such as vision, smell, taste and touch of a food sample through multi-modal data acquisition equipment, and performing standardization processing to obtain a sensory feature matrix; generating a sensory feature vector set by using a pre-established multi-modal feature extraction model; modeling each sensory dimension relationship by means of an attention layer in a deep learning network, and calculating to obtain an attention weight of each sensory feature; and performing weighted combination on the sensory feature vector set based on the weight to generate a comprehensive sensory feature basic representation. According to the method, the association among the multi-modal sensory characteristics is effectively modeled through an attention mechanism, and a scientific and efficient solution is provided for food sensory evaluation.

The invention discloses a visual tactile sensor fused with an optical waveguide and a preparation method thereof, and belongs to the technical field of intelligent tactile sensing. The visual tactile sensor comprises a fixed shell, and the top end of the fixed shell is open; the composite tactile sensing layer is arranged at the opening position of the fixed shell, and the composite tactile sensing layer is sequentially composed of a shading layer, a reflecting layer, a red-green spaced fluorescent layer, an elastic deformation layer and a supporting layer from top to bottom; and the optical lighting and sensing component comprises a blue LED light bar, an array light intensity sensor and a camera, the blue LED light bar and the array light intensity sensor are arranged on the two corresponding sides of the composite touch sensing layer respectively, and the camera is arranged at the bottom in the fixed shell. Through fusion of rapid and stable light intensity detection characteristics of optical waveguide sensing and pressure distribution detection capability of the visual tactile sensor, high-resolution space and texture identification and high-sensitivity mechanical response detection are simultaneously realized in a single sensor architecture.

The invention discloses a visual tactile sensor, an electronic skin and a robot, relates to the field of sensors, and is used for improving the comprehensiveness and sensitivity of stress detection. The sensor comprises a magnetic film covering the top surface and the side surface, and the magnetic film is prepared by mixing magnetic powder and silicone rubber; the support supports the magnetic thin film, a magnetic display card and a light source are arranged in the support, the visual touch collector collects magnetic induction data and development data of stress of the magnetic thin film, and then the multi-mode pressure detection model analyzes the stress position and / or the stress size. The stress detection sensitivity is high, comprehensive detection of the top face and the side faces can be achieved, and the detection precision is high.

The invention provides an industrial robot self-adaptive grabbing method based on multi-modal sensing, which comprises the following steps of: 1, acquiring multi-modal data of an object and an environment through a multi-modal sensing module; the multi-mode sensing module comprises at least two sensing modules of a visual sensing unit, a touch sensing unit, a force sensing unit and an auditory sensing unit; 2, preprocessing and feature extraction are carried out on the multi-modal data, feature information of different modals is fused through a multi-modal fusion algorithm, and comprehensive object description information is generated; 3, generating an optimal grabbing strategy based on the comprehensive object description information; wherein the grabbing strategy comprises the position and posture of a grabbing point, a grabbing path and grabbing force; and fourthly, the industrial robot executes the optimal grabbing strategy, the first step to the third step are repeated, and the grabbing strategy is adjusted in a self-adaptive mode till grabbing is completed. Stable and efficient grabbing can be achieved.

An embodiment of the utility model discloses a visual tactile sensor, which relates to the technical field of sensors and comprises a base, a flexible plate, an image acquisition device and a plane mirror. The flexible plate is arranged on the base, the image acquisition device is arranged on the base, the image acquisition device is located on one side of the flexible plate, and the plane mirror is obliquely arranged on the upper side of the flexible plate. The layout between the flexible plate and the image acquisition device can be changed through the plane mirror, the size of the visual tactile sensor in the height direction is greatly reduced, and the size of the visual tactile sensor is reduced. According to the scheme, through the innovative structural design of reflection imaging of the plane mirror, on the premise that the tactile perception precision is ensured, the problem that the size of a traditional visual tactile sensor is too large due to vertical imaging layout is solved, meanwhile, multiple technical requirements of environmental adaptability, cost controllability and the like are taken into consideration, and the visual tactile sensor is suitable for popularization and application. And better hardware support is provided for scenes such as refined operation and man-machine interaction of the robot.

The invention discloses a vision-based high-density three-dimensional force sensing array system and a three-dimensional force distribution identification method thereof. The system comprises a first force sensing layer, a second force sensing layer, a support force bearing layer, an LED lighting unit, a light-transmitting reference plate, a camera and a supporting shell box body. The second force sensing layer is a high-density array formed by independently distributed cylinder and hemisphere elastic structure units; and the light-transmitting reference plate, the LED lighting unit and the camera form an optical imaging assembly. When an external object is in contact with the surface of the array, each structural unit deforms independently, a camera captures a deformation image, features are extracted through an image processing algorithm, and three-dimensional force distribution of a contact area is solved according to a pre-calibrated image-mechanics mapping model. The system can reconstruct the three-dimensional contour form of a contact object in real time and perceive the force distribution characteristics of the contact object, and a high-precision and low-cost force touch perception solution is provided for the fields of robot touch recognition, human-computer interaction, safety monitoring and the like.

The invention discloses a small-size visual tactile sensor based on a UV mark, and relates to the field of sensors. The visual tactile sensor solves the problem that the existing visual tactile sensor cannot realize high-precision force and texture perception on the same sensor at the same time, independently or in a time-sharing manner. A UV fluorescence labeling layer which is transparent under visible light and develops under ultraviolet light is arranged on the contact surface of the elastomer; the composite light source module comprises a UV light source unit and an RGB light source unit which can be independently controlled; the UV light source or the RGB light source is turned on through time-sharing switching, so that the image acquisition module can acquire mark point displacement images used for calculating contact force distribution or photometric stereo images used for reconstructing contact surface textures in a time-sharing manner. According to the invention, the problem that a traditional sensor cannot realize lossless switching of force and texture sensing functions on a single device is fundamentally solved by using the invisibility characteristic of the UV mark and a dual-light-source time-sharing working mode. The method is used for robot touch sensing.

The invention discloses a tactile image domain migration method and device based on a multi-scale generative adversarial network, and the method comprises the steps: constructing the multi-scale generative adversarial network, and achieving the domain migration between a simulation tactile image and a real tactile image. The generator takes U-Net as a trunk, introduces a multi-scale stacking module, a multi-stage attention gate mechanism and a channel-space attention module, and improves the reconstruction capability of simulation image textures, illumination and contact areas. And the discriminator adopts a multi-scale discrimination structure to realize the discrimination of image authenticity and detail consistency. Through joint training of joint adversarial loss, loop consistency loss, contact area consistency loss, illumination balance loss and frequency domain loss functions, it is ensured that an output image is consistent with a real image in visual and semantic levels. The method can be widely applied to a robot vision-touch fusion perception task, and the migration performance and robustness of the perception model are remarkably improved under the condition of non-paired data.

The invention discloses a self-adaptive visual tactile sensor and an adjusting method, the self-adaptive visual tactile sensor comprises a base and a shell, a control panel, a camera and at least one light source assembly are fixedly arranged on the base, a contact assembly is fixedly arranged at the upper end of the shell, and the self-adaptive visual tactile sensor is characterized in that a light source rotation driving mechanism is arranged on the base, and a liquid lens is fixedly arranged on the base; the liquid lens is located between the camera and the contact assembly, the center of the liquid lens and the optical axis of the camera are located on the same straight line, and the local focal length of the liquid lens and the included angle between the light source assembly and the horizontal plane can be automatically adjusted through electric signals sent by the contact assembly. The system has the advantages that imaging of the camera is clear, and the imaging quality is ensured; moreover, the camera achieves secondary focusing through the liquid lens, so that the camera does not need to adjust the position of the lens for imaging of different object distances, the focusing distance between the contact assembly and the camera is greatly shortened, and the overall size of the visual tactile sensor is reduced.

The invention discloses a visual touch fusion object material identification method and device based on channel switching, and the method comprises the steps: obtaining a paired data sample which comprises an object and is composed of a visual image and a touch image, carrying out the preprocessing of the paired data sample, and constructing a training data set; constructing a visual touch fusion object material identification network, wherein the visual touch fusion object material identification network comprises a visual feature extraction sub-network, a touch feature extraction sub-network, a channel switching gating fusion module, a space attention module, a feature pyramid fusion module and a classification module; training the visual touch fusion object material identification network by using the training data set; and deploying the trained visual touch fusion object material identification network to a device end, and performing object material identification by using the trained visual touch fusion object material identification network to obtain an identification result. According to the invention, on the basis of the lightweight design of the model, the substance identification accuracy is obviously improved.

The invention discloses a three-dimensional elastic imaging method based on a visual tactile sensor, which belongs to the technical field of elastic imaging and comprises the following steps: system building and data processing, acquisition of contact deformation data acquired by the visual tactile sensor, forward simulation and inversion optimization. Calculating internal gradient information of a target object; deducing a derivative of a target function to Young modulus; calculating a partial derivative of resultant force to elastic distribution; calculating a gradient of the whole target function to Young modulus distribution; according to the method, the Young modulus distribution in the soft object can be accurately reconstructed under robot touch sampling, corresponding credibility indexes can be output, and the practicability is high.

The invention provides a multi-mode visual touch fusion sensor of a non-coaxial light path, which realizes multi-mode cooperation of a visual mode and a visual touch mode by performing non-coaxial arrangement on the optical axis of a camera and the central normal of a touch plane on a structural level; a virtual binocular system is constructed under the condition of a single camera by means of a free-form surface light splitting biprism, the number of devices and the assembling freedom degree are reduced, and therefore the integration degree and calibration consistency are improved; a semi-transparent and semi-reflective optical element is matched with an LED light-emitting unit time sequence and a control circuit to realize automatic and rapid mode switching, so that the long-term reliability is improved, and the switching service life is prolonged; through the collaborative design of the self-assembled metal nano reflective coating and the thin polyurethane packaging layer, a high-contrast and wear-resistant deformation image is obtained on the premise of not sacrificing the tactile sensitivity, and finally continuous acquisition of a target pose and contact parameters is realized. And the comprehensive requirements of accurate grabbing and touch closed-loop control on stable imaging, rapid switching, easy integration and high reliability are met.

The invention discloses a terrain classification and self-adaptive gait switching method for a sea-turtle-imitating amphibious robot based on visual tactile perception. The method comprises the following steps: establishing a robot model carrying a pressure sensor and a waterproof camera; the method comprises the following steps: acquiring tactile time sequence data and visual images in various terrains such as broken stones and sand beaches, and constructing a training data set; establishing a multi-modal convolutional neural network, and fusing features through dual-channel feature extraction and an adaptive attention mechanism; motion energy consumption and speed under each terrain are collected, and the optimal gait is determined; and constructing a gait switching finite-state machine in combination with terrain physical attribute classification and the optimal gait. According to the method, a sensing-decision-driving closed loop is formed, the terrain is sensed through visual touch, the gait is decided through a finite-state machine, the gait driving motion is executed, and through experimental verification, the self-adaptive gait can improve the moving efficiency and speed, and the amphibious environment adaptability of the robot is enhanced.