451 results about "Robot vision" patented technology

The invention relates to the field of image processing, and particularly discloses a welded pipe surface defect detection method based on robot visual inspection. The method comprises the steps that a robot carries a binocular camera and an annular LED light source and moves at a constant speed in the axial direction of a welded pipe to collect orthographic and inclined views, and a three-dimensional point cloud is constructed; and establishing a parameterized mapping function based on the point cloud, and converting the 3D coordinate into a 2D expansion surface coordinate. In the convolutional neural network, a first layer is inserted into a spatial transformation network to correct distortion of the expanded image, deformable convolution is adopted to extract edge, local deformation and specific defect response features, and standard convolution is combined to extract global features; and fusing multi-scale features and adding an attention mechanism to improve the weight of a defect region, and outputting a defect category and a bounding box offset after generating a candidate box. The method effectively solves the problems of stretching, deformation and defect distortion of welded pipe curved surface imaging, reduces the imaging difference of the same defect, and remarkably improves the defect positioning precision and recognition accuracy.

The invention relates to the technical field of robot visual perception and motion control, and discloses a motion control method and device based on physical reference, equipment and a medium, and the method comprises the steps: obtaining instruction information, a multi-view image and movable assembly pose information; processing the multi-view image according to the instruction information to generate target segmentation information; generating a scale normalization point cloud and a model estimation baseline; determining a physical reference baseline and generating a scale calibration factor; converting the scale normalization point cloud into a physical space point cloud by using a scale calibration factor; extracting a three-dimensional relative position of the target object relative to the movable component in combination with the target segmentation information; an action instruction is generated based on the multi-modal input. According to the method, physical scale alignment of the point cloud is realized through physical reference baseline calibration, so that a visual reconstruction result has real space significance, an accurate action instruction is generated, and the robot space understanding and operation precision is improved.

The invention discloses a joint denoising method for robot visual motion prediction, and the method comprises the steps: constructing a unified generative model through fusing an image and a depth map collected by a depth camera, motion data collected by CAN line communication of a Piper mechanical arm, and a tactile image collected by a Gelsight Mini tactile sensor; the method comprises two steps of data acquisition and input coding, and joint denoising and generation: firstly, multi-modal data are coded into low-dimensional potential representation, and then future images, depth maps, tactile data and robot actions are cooperatively predicted through a joint denoising framework based on Transform. A mask self-attention mechanism is innovatively introduced, information interaction between modes is dynamically adjusted, action generation is guided through tactile feedback, and the force control precision is improved. The model adopts a de-noising diffusion probability loss function to jointly optimize multi-modal prediction, so that the output consistency is ensured. According to the method, the robustness and the accuracy of flexible operation of the robot are remarkably improved.

The invention discloses a digital twin interaction control system based on robot vision and tactile fusion, and the system comprises a data collection module which is used for collecting visual image data and tactile sensing data; the preprocessing module is used for preprocessing; the multi-mode perception modeling module is used for executing spatial perception modeling and tactile state modeling; the fusion state vector construction module is used for generating a fusion state vector; the digital twinborn mapping module is used for constructing a digital twinborn body; the simulation interaction control module is used for executing interactive action control and generating a control instruction parameter group; the action execution and feedback module is used for controlling the robot to execute the actual action and updating the fusion state vector; and the dynamic evaluation and self-adaptive adjustment module is used for dynamically optimizing and adjusting the fusion state vector. Visual and touch information of the robot is fused, a digital twinning synchronous control mechanism is constructed, and the interactive optimization method which is fast in operation response, high in execution precision and stable in control process is achieved.

The invention relates to the technical field of tumor interventional operations, and discloses a visual navigation method based on a tumor interventional operation robot. The method comprises the following steps: acquiring real-time medical image data of a tumor area containing multi-modal imaging information so as to comprehensively present anatomical details; and performing three-dimensional reconstruction on the image data to generate a tumor area three-dimensional anatomical structure model capable of visually displaying a space structure. Key anatomical feature points are extracted based on the model, space coordinates are calculated, a surgical robot intervention path is planned according to the coordinates, and an initial navigation track is generated; and continuously collecting real-time pose data of the robot in an operation, dynamically matching the real-time pose data with the initial navigation trajectory, adjusting motion parameters according to a matching result, and generating a corrected navigation instruction. The method can reflect the intraoperative anatomy condition in real time, dynamically optimize the path, solve the problems that traditional navigation depends on preoperative static images and lacks real-time adjustment, reduce operative complications and improve the treatment effect of patients.

The invention relates to a leather material cutting robot and a cutting method thereof, and belongs to the technical field of material cutting, the leather material cutting robot comprises a human-computer interaction interface, a CAD / CAM system, a multi-mode visual perception unit, a central processing system, a dynamic compensation cutting unit and a quality closed-loop system, and the multi-mode visual perception unit comprises a high-resolution multispectral imaging module and a 3D scanning module. According to the method, the problems that a traditional leather material cutting robot visual recognition system cannot recognize leather thickness changes, material waste is easily caused by subcutaneous defects, natural curling deformation of leather is difficult to adapt, and the edge precision error is large are solved, the rejection rate can be further reduced through multispectral and 3D defect recognition, and the production efficiency is improved. Three-dimensional dynamic compensation enables edge errors to be further reduced, and real-time force control further reduces the occurrence rate of crush damage.

The invention relates to the field of robot vision positioning, and discloses an intelligent hand-eye calibration and self-adaptive correction system and method. According to the method, a mechanical arm is controlled to drive a camera to collect multi-modal calibration data, a convolutional neural network is utilized to identify a calibration plate mark point, and a three-dimensional coordinate under a camera coordinate system is calculated in combination with depth information; meanwhile, on the basis of an encoder and torque data, flexible deformation of the mechanical arm is compensated through a kinematic model and a self-adaptive rigidity model, and three-dimensional coordinates under a base coordinate system are obtained. And obtaining a hand-eye transformation matrix by solving a transformation relation between the two point sets. And repeatedly calibrating before and after operation, inputting the difference of the two transformation matrixes into the fault diagnosis neural network, outputting fault type probability distribution, and generating a maintenance strategy. According to the invention, high-precision hand-eye calibration, automatic compensation of flexible deformation and intelligent diagnosis of system state change can be realized, so that the long-term precision and reliability of a visual positioning system are improved.

The invention provides a robot vision object semantic understanding and posture generation method based on a large model, and belongs to the technical field of image processing. Comprising the following steps: S1, inputting an image and a 3D model; s2, object detection; s3, multi-modal feature alignment is carried out; s4, diffusion model sampling; s5, performing geometric screening; s6, carrying out NeRF (New Random Field) morphological modeling; s7, optimizing the optical flow; s8, joint loss calculation; s9, confidence coefficient analysis; and S10, outputting the attitude junction. According to the invention, through an innovative single-view rendering-optical flow optimization closed loop strategy, the calculation overhead is significantly reduced and the estimation precision is improved. According to the method, the algorithm performance in a complex scene is remarkably improved through multi-scale feature fusion and a confidence decomposition strategy.

The invention relates to the technical field of agricultural picking robot visual perception, and discloses a strawberry fruit detection and picking key point positioning method and model based on deep learning. The method comprises the following steps: acquiring strawberry RGB-D images to construct a data set; on the basis of a YOLOv8-Pose architecture, global feature extraction is enhanced by constructing a C2f-MAMBA module of a fusion state space model, an HWD module is introduced to retain detail features, and a Faster Net Block is used for weight reduction, so that an STRAW-MAMBA model is constructed; outputting coordinates of five key points such as a fruit bounding box and a shear point by the model, and performing geometric compensation on a shielding point; and finally, calculating three-dimensional space coordinates of the key points in combination with depth information. According to the method, a C2f-MAMBA module based on Mama is innovatively introduced, global feature association is effectively modeled, a multi-scale attention mechanism is combined, the defect that long-range dependence is difficult to capture in a traditional CNN is overcome, and the detection and positioning precision of strawberry fruits and picking key points thereof in a complex background is remarkably improved.

The invention provides a binocular vision global positioning system and method for a mobile robot, and relates to the technical field of robot vision navigation, and the method comprises the steps: carrying out the adaptive exposure compensation of binocular images of a left camera and a right camera, extracting edge contour feature points irrelevant to illumination, and generating a high-quality depth feature map through the combination of parallax calculation; meanwhile, feature units are constructed according to the inflection points, so that the global position of the mobile robot is determined. Therefore, the problem that the global positioning precision is reduced due to unstable visual feature extraction caused by rapid change of illumination intensity of the mobile robot in a multi-region switching scene can be solved to a certain extent.

Apparatuses, systems, and techniques are disclosed for controlling a robot to execute a task. In at least one embodiment, a current image of the robot in an environment and a text describing the task are obtained. A future image of the robot in the environment is predicted based on the current image and the text. Subsequently, one or more actions are predicted based on the current image, the future image, and the text. The one or more actions can move the robot from a first state corresponding to the current image to a second state corresponding to the future image. The robot executes the sequence of actions to move in the environment.

The invention discloses a distribution network hot-line work robot pose estimation system and method, and belongs to the technical field of robot visual perception. The system comprises an input preprocessing module which is used for carrying out noise reduction and enhancement processing on an RGB-D image; the shared feature extraction module is used for extracting multi-scale universal features based on a lightweight convolution architecture; the 6D pose estimation module is used for processing the image based on the neural implicit field to obtain a pose estimation result; and the joint optimization module is used for realizing detection and pose estimation shared feature extraction through cooperation of a multi-task loss function and a pose estimation result. The system adopts adaptive median filtering and homomorphic filtering to eliminate noise and uneven illumination, and bilateral filtering optimizes a depth map; a bottleneck structure and cavity convolution are introduced into feature extraction, and a rank enhancement linear attention module is embedded; according to the pose estimation, a geometric field and an appearance field are modeled through a neural implicit field, and pose hypotheses are generated and optimized. According to the method, the problems of low pose estimation precision and poor real-time performance in the distribution network live working environment are solved, and the working safety and efficiency of the robot are improved.

The invention is suitable for the technical field of artificial intelligence, and provides a robot visual navigation optimization system based on edge computing, the system comprises a visual sensor module, an edge computing unit, a navigation algorithm module, a communication module and a robot executor, the visual sensor module is used for collecting high-definition images or video streams, and the edge computing unit is used for processing the high-definition images or video streams; the edge calculation unit is responsible for locally executing a visual navigation algorithm, the navigation algorithm module is deployed on the edge calculation unit, and the navigation algorithm module is used for executing a visual odometer, synchronous positioning and map construction, environment perception, obstacle detection, local path planning and motion control; and the communication module is used for carrying out model updating, task scheduling and non-real-time data uploading with a cloud. Complex visual data processing and navigation decision are sunk to the robot body or near-end edge equipment, so that the navigation precision and the response speed of the robot in complex and diversified environments are remarkably improved, and the energy consumption efficiency is optimized.

The invention relates to a calibration method of a binocular camera and a laser radar based on a hollow ellipse calibrator, and belongs to the field of robot vision. Hamming code graticule marks are respectively arranged at four corners of one side face of a square aluminum plate of the hollow elliptical calibrator in the clockwise direction, an elliptical through hole is formed in the middle of the hollow elliptical calibrator, and the center of the elliptical through hole coincides with the center of the square aluminum plate; and the long axis of the elliptical through hole is positioned in the vertical direction. A hardware system of the calibration method comprises a binocular camera, an upper computer, a 16-line laser radar and a hollow ellipse calibrator. An image processing program, a three-dimensional point cloud processing program and a mathematical calculation program are arranged in the upper computer; the calibration method comprises the following operation steps: (1) carrying out image feature recognition on the hollow ellipse calibrator; (2) carrying out point cloud feature recognition on the hollow ellipse calibrator; (3) calculating external parameters of the camera and the 16-line laser radar; and (4) determining a calibration result. According to the calibration method, the calibration accuracy and robustness are improved, and the calibration method is suitable for practical application.

According to the robot vision calibration device and method, through collaborative design of the industrial personal computer, the vision system, the robot, the calibration jig and the calibration plate, the problems caused by complex structure, low calibration precision and poor adaptability in the prior art are solved. A floating mechanism of the calibration jig is combined with a displacement sensor to realize multi-degree-of-freedom dynamic displacement compensation, so that mechanical positioning errors are remarkably reduced; the conical surface guide structure and the reference hole layout of the calibration plate optimize the stability of the positioning reference, and ensure the accurate embedding of the calibration needle. The flexible installation of the visual system adapts to different scene requirements. According to the calibration method, the generation of a high-precision visual calibration matrix is realized through the processes of multi-reference hole coordinate recording, calibration ring coordinate calculation and visual image acquisition and matching. According to the whole scheme, through a dynamic compensation mechanism and intelligent process design, the calibration precision, efficiency and robustness are improved, meanwhile, the operation complexity is reduced, and reliable technical support is provided for an industrial automation scene.

The invention discloses a three-dimensional calibration automatic system suitable for a six-axis physiotherapy robot, and relates to the technical field of robot vision calibration and automatic control, and the system comprises an equipment communication management module which automatically identifies and distributes connection equipment through an intelligent detection technology, and outputs an equipment control instruction; the data acquisition and processing module is used for identifying 13-point feature pixels through RGB information, performing deep matching, calculating 13-point theoretical feature three-dimensional information and outputting an optimized calibration matrix MB2R in combination with a transformation relation between the camera and the tail end of the mechanical arm and a base coordinate transformation relation; and the result verification and backup uploading module is used for converting the relative position of the three-dimensional feature platform and the robot, acquiring three-dimensional information of an evaluation point, controlling the mechanical arm to move to an optimization point position, and calculating a pixel difference value and a height difference value. The method has the advantages that full-automatic millimeter-level calibration precision is achieved through a collaborative structure of multi-protocol heterogeneous equipment intelligent integration, iterative optimization algorithm fitting and closed-loop double-index verification.

The invention discloses a robot vision system construction and control method, and the method comprises the following steps: collecting environment image data and robot motion state data, carrying out the real-time preprocessing of the environment image data, generating standardized image data, and carrying out the real-time monitoring of the environment through setting a dynamic environment sensing module. When real-time grabbing operation guided by robot vision is carried out, an environment light interference real-time judgment mechanism is established, differential compensation strategies are set for different working condition scenes, the stability of visual data collection under the complex illumination condition is guaranteed, and meanwhile compensated light parameters are synchronized to an image processing unit; the problem of feature recognition distortion caused by abnormal environmental reflection can be reduced in real time, the accuracy of robot target positioning is ensured, the track error of visual guidance is reduced, and the completeness of a high-precision assembly task is ensured when it is detected that the in-place posture offset exceeds a safety threshold value.

The application discloses a mechanical arm cable assembly method and system, and relates to the technical field of robot vision perception and control, and comprises the following steps: acquiring point cloud data of a working environment; fitting a cable center line according to the point cloud data and determining a relative pose matrix of a cable terminal and a target slot; determining an initial alignment attitude matrix of a mechanical arm end according to the relative pose matrix, so as to obtain initial joint angles of the mechanical arm and use the initial joint angles as a starting point of an assembly action; after selecting a grabbing point on the cable center line and planning an insertion path, entering an insertion stage from the starting point; according to detection of an insertion force and an insertion direction, performing error calculation and compensation until the insertion force change rate and the end pose error meet a set condition, and then determining that the assembly is completed. The application realizes self-perception, self-judgment and self-adjustment flexible assembly, improves assembly precision and robustness, and is suitable for automatic grabbing, guiding and insertion operation of flexible cables in industrial automation production.

The invention relates to the technical field of underwater robot control, and particularly discloses a visual servo scanning track control system for a deepwater ROV intelligent electromagnetic detection robot, which synchronously acquires visual, acoustic and inertial motion data of an underwater robot and prior geometric data of a to-be-detected structure; visual sparse light spot features and acoustic contour and abnormal region features are extracted, and credibility evaluation is carried out; by establishing a unified state vector and fusing multi-source data and geometric constraints, the accurate pose, the motion state and the perception uncertainty measurement of the robot relative to the underwater structure are estimated in real time; according to the uncertainty measurement, a local expected trajectory used for precise tracking or pose stabilization is generated in a self-adaptive mode; a motion control instruction is calculated through a control law capable of automatically adjusting gain according to uncertainty, and closed-loop control is achieved; according to the invention, continuous, self-adaptive and high-precision execution of detection operation under complex working conditions is realized.

The invention discloses a robot visual guiding and grabbing system for photovoltaic glass stacking, and the system comprises a multi-mode visual data collection module which is used for synchronously obtaining two-dimensional image data and three-dimensional depth data of photovoltaic glass; and the environment adaptive sensing module is used for preprocessing the two-dimensional image data and the three-dimensional depth data to eliminate or weaken the influence of illumination variation, surface reflection and noise on the image quality, and outputting enhanced image data and calibration depth data. According to the robot vision guiding and grabbing system for photovoltaic glass stacking, by introducing the multi-mode vision data acquisition module, two-dimensional image data and three-dimensional depth data of photovoltaic glass are synchronously obtained, and the original data are subjected to reflection removal, denoising and enhancement processing in combination with the environment self-adaptive sensing module; the problem that the image acquisition quality of a traditional visual system is unstable in a high-reflection and complex illumination environment is effectively solved.

The invention discloses a mechanical arm cable assembly method and system, and relates to the technical field of robot visual perception and control, and the method comprises the steps: obtaining the point cloud data of a working environment; fitting a cable center line according to the point cloud data and determining a relative pose matrix of the cable terminal and the target slot; an initial alignment attitude matrix of the tail end of the mechanical arm is determined according to the relative pose matrix, and an initial joint angle of the mechanical arm is obtained through solving and serves as a starting point of the assembly action; after a grabbing point is selected on the center line of the cable and a plug-in mounting path is planned, starting to enter a plug-in mounting stage from a starting point; and error calculation and compensation are carried out according to detection of the insertion force and the insertion direction, and it is judged that assembly is completed until the insertion force change rate and the tail end pose error meet set conditions. Self-sensing, self-judging and self-adjusting flexible assembling is achieved, the assembling precision and robustness are improved, and the flexible cable automatic grabbing, guiding and inserting device is suitable for automatic grabbing, guiding and inserting operation of flexible cables in industrial automatic production.

The embodiment of the invention relates to the technical field of bionic robots, in particular to a bionic robot vision collaboration method based on multiple vision modules, a robot vision device, a vision unit device and a robot. The method comprises the steps that S1, a task instruction is received and analyzed, and task semantics are obtained; s2, acquiring main view image data and extracting low-level visual features; s3, searching a bionic attention weight mapping table to obtain advanced visual feature requirements and initial feature weights; s4, generating an initial dynamic attention thermodynamic diagram, and determining a high-attention area and a secondary-attention area; s5, acquiring high-resolution image data, and updating low-level visual features of the front view image data; and S6, performing weighted fusion, updating the dynamic attention thermodynamic diagram in real time, and forming a perception-action closed loop. According to the method, by simulating a visual attention mechanism of task-driven selective focusing from top to bottom and saliency perception from bottom to top according to tasks, the improvement of the visual collaborative bionic degree of the robot is realized.

The invention discloses a tomato picking point positioning and feasibility analysis system based on RGB-D attitude estimation, and belongs to the technical field of agricultural robot visual perception, and the system comprises an RGB-D data collection module which is used for synchronously collecting an RGB image and a depth image of a tomato cluster; a data preprocessing module; a multi-scale feature extraction and fusion module; the multi-task prediction module comprises a detection and key point prediction head, a picking point depth regression prediction head and a picking feasibility analysis module which are arranged in parallel; the tomato cluster three-dimensional picking system has the advantages that synchronous output of tomato cluster structure attitude information, picking point three-dimensional positions and picking feasibility judgment is achieved, tomato cluster attitude sensing stability and picking point three-dimensional positioning precision are improved, and the tomato cluster three-dimensional picking system is high in practicability and high in practicability. And the robot is endowed with an intelligent picking decision-making capability, and the operation success rate of the tomato picking robot is improved.

The invention discloses a migration method and device from simulation to a real image for an unmanned system, and the method comprises the steps: constructing a double-flow architecture comprising a convolution encoder and a visual Mama encoder, wherein the convolution encoder is used for extracting the local texture information of an image, and the visual Mama encoder employs a visual state space model to capture the global context information of the image; and the two features are input into a decoder after channel dimension fusion, and a real style image after style migration is generated. In order to improve the unsupervised learning ability of the model, a cyclic consistency training framework is introduced, training can be completed under the condition of no paired image samples, and the image structure consistency is kept. According to the method, the sense of reality and the structural fidelity of a simulation image are effectively improved, and compared with an existing method, the method shows lower distortion and higher generalization ability in an image translation task. The method can be widely applied to simulation data field adaptation and training sample enhancement tasks in the fields of automatic driving, virtual simulation, robot vision and the like.

The invention relates to the technical field of robot visual guidance, and discloses a visual guidance robot material box grabbing module, which comprises a mechanical coarse positioning and state sensing unit for physically clamping a material box and collecting physical state data; a prediction type pre-positioning and dynamic reference generation unit receives the data, calls a historical deviation database, and generates a prediction target position and a temporary standard reference in parallel; the nested visual search and calibration unit moves to a prediction position and searches to obtain a complete feature image; the accurate deviation calculation and fine adjustment unit compares the image with a reference, calculates a final residual deviation and generates an accurate grabbing position; and the closed-loop learning and data feedback unit executes capturing and feeds back the final residual deviation to the historical deviation database. According to the method, a predicted target position and a dynamic temporary reference are established by using a physical state data vector and a historical deviation database, and parallel compensation of system errors and mechanical errors is realized.

The invention discloses a tire pattern block double-process precision machining device in the technical field of machine vision. The tire pattern block double-process precision machining device comprises a robot. The visual control system comprises an industrial personal computer and visual upper computer software; the binocular camera is mounted on the fixed bracket and is used for realizing high-precision reconstruction of the curved surface of the pattern block and high-precision positioning of a back hole; the light source assembly comprises a customized arc-shaped light source, a customized support and a light source controller, and the light source assembly comprises a customized arc-shaped curved surface light source which is installed at the top of the support and used for lighting and installed on the side face of the support and used for supplementing light; the rotary workbench comprises a circular rotary platform driven by a motor and a pattern block clamping mechanism; the rotary workbench is used for rotating the tire pattern blocks placed at the designated positions; the machining device comprises an electric spindle and a controller of the electric spindle. The operation table comprises a customized electric cabinet and a touch screen mounted on the surface of the electric cabinet; the tool setting device is mounted on the outer side of the rotary table and used for compensating tool changing errors; according to the method, the limitation on the precision machining effect caused by insufficient consistency of binocular vision in depth estimation is overcome, the pattern block chamfering and milling groove machining task with the precision requirement being 0.2 mm or above is finally completed, and meanwhile, the consistency of the machining quality of the tire pattern block back hole is guaranteed.

The invention discloses a robot grabbed target recognition and positioning method and system based on a knowledge graph, and the method comprises the steps: constructing a triple knowledge graph containing object appearance features and grabbed point pose parameters through multi-source data integration, and achieving a structured node network; environment perception data of a robot vision module is obtained, vector representation of a target object is determined through processing, if the similarity exceeds a threshold value, a matching node is inquired through the entity linking technology, recognition ambiguity is solved, and an accurate result is obtained; based on a result, retrieving pose parameters in the knowledge graph, performing coordinate conversion to generate a fine-grained grabbing sequence, combining real-time visual feedback, adopting a dynamic adjustment algorithm to optimize a path, and updating the knowledge graph through simulation evaluation to adapt to a new scene. According to the method, the core function of the knowledge graph in grabbing optimization is highlighted, the accuracy, robustness and adaptive capacity of robot operation are improved, and the method is suitable for the fields of industrial automation and the like.

The invention discloses a double-arm cooperative bolt alignment method and system based on hybrid visual servo, and relates to the field of robot visual servo and cooperative control, and the method comprises a visual perception module which is used for obtaining part space pose information based on first image information of a part where a target bolt is located; according to the second image information of the target bolt, obtaining a two-dimensional image coordinate and space depth information of a bolt angular point of the target bolt; the hybrid visual servo control module is used for generating a motion control instruction of the main mechanical arm by utilizing a staged visual alignment control strategy based on the spatial pose information of the part and the two-dimensional image coordinates and spatial depth information of the angular points of the bolts; and the double-arm cooperative control module is used for generating a cooperative motion instruction of the slave mechanical arm based on a cooperative control algorithm in the process that the master mechanical arm carries out visual servo alignment according to the motion control instruction. According to the method, the bolt alignment task under the complex working condition can be completed in a self-adaptive mode, and the intelligent level of overhauling operation is improved.

This invention discloses a deep learning-based robot visual recognition and decision-making control method, belonging to the field of control system technology. It includes constructing a thought decision tree based on input commands and real-time visual images, obtaining the linear relationship between the middle-level leaf nodes and the top leaf nodes, and using this relationship to associate current visual features with task intent. By constructing a multi-level thought decision tree structure, combined with multi-modal feature fusion and a deep learning engine, this invention achieves deep fusion and unified representation of multi-source heterogeneous data. Furthermore, by constructing a fusion search engine and semantic alignment mechanism, it effectively integrates text commands with visual images and other modal information, enabling accurate parsing of task intent and collaborative processing of environmental perception, and improving the real-time response and decision-making accuracy of service robots for complex tasks.

The invention discloses a tactile image domain migration method and device based on a multi-scale generative adversarial network, and the method comprises the steps: constructing the multi-scale generative adversarial network, and achieving the domain migration between a simulation tactile image and a real tactile image. The generator takes U-Net as a trunk, introduces a multi-scale stacking module, a multi-stage attention gate mechanism and a channel-space attention module, and improves the reconstruction capability of simulation image textures, illumination and contact areas. And the discriminator adopts a multi-scale discrimination structure to realize the discrimination of image authenticity and detail consistency. Through joint training of joint adversarial loss, loop consistency loss, contact area consistency loss, illumination balance loss and frequency domain loss functions, it is ensured that an output image is consistent with a real image in visual and semantic levels. The method can be widely applied to a robot vision-touch fusion perception task, and the migration performance and robustness of the perception model are remarkably improved under the condition of non-paired data.