577 results about "Robot vision" patented technology

The invention belongs to the technical field of image processing, and particularly relates to an object six-degree-of-freedom pose estimation method based on three-dimensional geometric information registration, which comprises a target detection module, a three-dimensional point cloud generation model and a geometric information registration pose estimation module. According to the method, geometric reconstruction, a geometric information high-precision registration technology and an object pose estimation task are combined, a geometric reconstruction module introduces object geometric feature extraction and feature decoding to realize deep understanding and modeling of an algorithm on object geometric features, the whole process accords with geometric disorder, and the accuracy of object pose estimation is improved. And the accuracy and scene adaptability of the algorithm are further improved. According to the method, accurate estimation of the six-degree-of-freedom pose of the object can be achieved, and powerful support is provided for the fields of industrial automation, robot vision, AR / VR, automatic driving, medical robots and the like.

The invention discloses an AI-based neurosurgery auxiliary robot visual positioning system, and relates to the field of visual positioning, which comprises the steps of deploying and initializing structured light scanning equipment and near-infrared imaging equipment, carrying out multi-modal image acquisition on a surgical area, and carrying out standardization processing, space-time alignment and fusion on the acquired image. A multi-modal visual acquisition and preprocessing module, a three-dimensional tissue model construction and registration module, a visual-anatomical feature recognition and extraction module, an AI auxiliary positioning and path optimization module, an intraoperative dynamic perception and feedback control module and a target position confirmation and instruction output module are constructed. According to the method, high-precision identification and dynamic modeling of a brain tissue structure are realized, real-time identification, path planning and position correction can be carried out on a key anatomical structure in an operation process, the precision, the intelligent level and the intra-operation response capability of neurosurgery operation are remarkably improved, the operation risk is effectively reduced, and the positioning reliability and the automatic control efficiency are improved.

The invention relates to the field of image processing, and particularly discloses a welded pipe surface defect detection method based on robot visual inspection. The method comprises the steps that a robot carries a binocular camera and an annular LED light source and moves at a constant speed in the axial direction of a welded pipe to collect orthographic and inclined views, and a three-dimensional point cloud is constructed; and establishing a parameterized mapping function based on the point cloud, and converting the 3D coordinate into a 2D expansion surface coordinate. In the convolutional neural network, a first layer is inserted into a spatial transformation network to correct distortion of the expanded image, deformable convolution is adopted to extract edge, local deformation and specific defect response features, and standard convolution is combined to extract global features; and fusing multi-scale features and adding an attention mechanism to improve the weight of a defect region, and outputting a defect category and a bounding box offset after generating a candidate box. The method effectively solves the problems of stretching, deformation and defect distortion of welded pipe curved surface imaging, reduces the imaging difference of the same defect, and remarkably improves the defect positioning precision and recognition accuracy.

The invention relates to the technical field of welding automation and robot visual perception, and discloses a robot intelligent welding method and system.The method comprises the steps that a three-dimensional point cloud image is constructed through a laser triangulation method; filtering and enhancing the point cloud image, and extracting a weld point cloud set by adopting a U-Net semantic segmentation model; carrying out space path identification and attitude regression based on the set to obtain a track element of a position-attitude pair; fitting a path by adopting a cubic B-spline curve and interpolating to generate a continuous welding track; and in the welding process, visual feedback is combined, and proportional differential control logic is introduced for closed-loop track error compensation. Compared with the technical problem that in the prior art, weld joint path recognition and accurate tracking cannot be stably achieved under complex structures such as curved surfaces and variable cross sections, a perception control closed loop based on visual servo is constructed, high-precision automatic welding of the complex weld joint structures is achieved, and the track fitting precision is improved.

The invention discloses a robot vision-inertia SLAM method and device and a medium, and belongs to the technical field of computer vision and robot navigation. The method comprises the following steps: synchronously acquiring images and inertial data through a robot binocular camera and an IMU, performing feature enhancement on an original image in combination with a pre-trained deep learning model, introducing an adaptive brightness compensation mechanism and designing an image light supplementing module based on a generative adversarial network (GAN), recovering low-illumination image details, and improving the image quality of a low-illumination area; an entropy-based adaptive dynamic interference rejection algorithm is provided, and dynamic interference feature points are rejected in combination with IMU (Inertial Measurement Unit) data; a low-rank approximate improved graph optimization algorithm is adopted, and global map construction and pose optimization are accelerated; and through entropy-based nonlinear dynamic smoothing coefficient adjustment, the track stability is improved. According to the method, the positioning precision and robustness in a low-light environment are improved, the calculation efficiency is improved, and dynamic interference is effectively resisted.

The invention discloses an automatic dovetail welding seam grinding control method based on robot visual positioning, and relates to the technical field of visual positioning. The method comprises the steps that after welding is completed, a dovetail welding seam image is collected and converted into a three-dimensional coordinate through a vision algorithm, and a three-dimensional point cloud is generated; smooth interpolation is performed on the three-dimensional point cloud through a B spline curve method to obtain a parameterized curve, and a control point set is optimized through a genetic algorithm to generate a global optimal polishing path; the polishing robot executes a task according to a path, a tail end sensor collects a real-time path and calculates a deviation value with a global path, and when the deviation value exceeds a preset threshold value, inverse kinematics is triggered to solve and correct the path; the hardness of the dovetail weld is measured through laser-induced breakdown spectroscopy, a comprehensive hardness value is obtained in combination with a matrix hardness database, meanwhile, a visual sensor collects the surface state, and polishing process parameters are dynamically adjusted according to the surface state; after the task is completed, the welding seam angle and flatness are detected. According to the method, the optimal path is generated through visual positioning, and automatic grinding of the dovetail welding seam is achieved.

The invention relates to the technical field of robot visual perception and motion control, and discloses a motion control method and device based on physical reference, equipment and a medium, and the method comprises the steps: obtaining instruction information, a multi-view image and movable assembly pose information; processing the multi-view image according to the instruction information to generate target segmentation information; generating a scale normalization point cloud and a model estimation baseline; determining a physical reference baseline and generating a scale calibration factor; converting the scale normalization point cloud into a physical space point cloud by using a scale calibration factor; extracting a three-dimensional relative position of the target object relative to the movable component in combination with the target segmentation information; an action instruction is generated based on the multi-modal input. According to the method, physical scale alignment of the point cloud is realized through physical reference baseline calibration, so that a visual reconstruction result has real space significance, an accurate action instruction is generated, and the robot space understanding and operation precision is improved.

The invention discloses a joint denoising method for robot visual motion prediction, and the method comprises the steps: constructing a unified generative model through fusing an image and a depth map collected by a depth camera, motion data collected by CAN line communication of a Piper mechanical arm, and a tactile image collected by a Gelsight Mini tactile sensor; the method comprises two steps of data acquisition and input coding, and joint denoising and generation: firstly, multi-modal data are coded into low-dimensional potential representation, and then future images, depth maps, tactile data and robot actions are cooperatively predicted through a joint denoising framework based on Transform. A mask self-attention mechanism is innovatively introduced, information interaction between modes is dynamically adjusted, action generation is guided through tactile feedback, and the force control precision is improved. The model adopts a de-noising diffusion probability loss function to jointly optimize multi-modal prediction, so that the output consistency is ensured. According to the method, the robustness and the accuracy of flexible operation of the robot are remarkably improved.

The invention discloses a robot visual identification decision control method based on deep learning, and relates to the technical field of control systems. Comprising the steps of constructing a thinking decision tree according to an input instruction and a real-time visual image, and obtaining a linear relationship between a middle-layer leaf node as a main body and a top-layer leaf node for realizing association between a current visual feature and a task intention. According to the method, deep fusion and unified representation of multi-source heterogeneous data are realized by constructing a multi-layer thinking decision tree structure and combining multi-modal feature fusion and a deep learning engine, and text instructions, visual images and other different modal information are effectively integrated by constructing a fusion search engine and a semantic alignment mechanism, so that the multi-source heterogeneous data fusion and unified representation are realized. Cooperative processing of accurate analysis of task intentions and environment perception is realized, and real-time response and decision accuracy of the service robot to complex tasks are improved.

The invention discloses a digital twin interaction control system based on robot vision and tactile fusion, and the system comprises a data collection module which is used for collecting visual image data and tactile sensing data; the preprocessing module is used for preprocessing; the multi-mode perception modeling module is used for executing spatial perception modeling and tactile state modeling; the fusion state vector construction module is used for generating a fusion state vector; the digital twinborn mapping module is used for constructing a digital twinborn body; the simulation interaction control module is used for executing interactive action control and generating a control instruction parameter group; the action execution and feedback module is used for controlling the robot to execute the actual action and updating the fusion state vector; and the dynamic evaluation and self-adaptive adjustment module is used for dynamically optimizing and adjusting the fusion state vector. Visual and touch information of the robot is fused, a digital twinning synchronous control mechanism is constructed, and the interactive optimization method which is fast in operation response, high in execution precision and stable in control process is achieved.

The invention relates to the technical field of robot visual guidance, in particular to an out-of-order target recognition and positioning method based on binocular laser three-dimensional scanning imaging, which comprises the following steps of: performing laser scanning on an object in a target area, and synchronously acquiring laser stripe images by adopting a binocular camera to obtain a target area image sequence; extracting a laser stripe center line of the target area image sequence to obtain auxiliary features; performing three-dimensional point cloud reconstruction on the auxiliary features to obtain original point cloud data; compensating the original point cloud data to obtain fused point cloud data; on the basis of a spherical multi-view sampling and positioning algorithm framework, off-line model point cloud features are constructed, and object recognition and pose matching are carried out on fused point cloud data; and visualizing an object identification and pose matching result. The method has the characteristics of stable identification and positioning results, high applicability in complex scenes, high identification accuracy and the like.

The invention relates to the technical field of tumor interventional operations, and discloses a visual navigation method based on a tumor interventional operation robot. The method comprises the following steps: acquiring real-time medical image data of a tumor area containing multi-modal imaging information so as to comprehensively present anatomical details; and performing three-dimensional reconstruction on the image data to generate a tumor area three-dimensional anatomical structure model capable of visually displaying a space structure. Key anatomical feature points are extracted based on the model, space coordinates are calculated, a surgical robot intervention path is planned according to the coordinates, and an initial navigation track is generated; and continuously collecting real-time pose data of the robot in an operation, dynamically matching the real-time pose data with the initial navigation trajectory, adjusting motion parameters according to a matching result, and generating a corrected navigation instruction. The method can reflect the intraoperative anatomy condition in real time, dynamically optimize the path, solve the problems that traditional navigation depends on preoperative static images and lacks real-time adjustment, reduce operative complications and improve the treatment effect of patients.

The invention relates to a leather material cutting robot and a cutting method thereof, and belongs to the technical field of material cutting, the leather material cutting robot comprises a human-computer interaction interface, a CAD / CAM system, a multi-mode visual perception unit, a central processing system, a dynamic compensation cutting unit and a quality closed-loop system, and the multi-mode visual perception unit comprises a high-resolution multispectral imaging module and a 3D scanning module. According to the method, the problems that a traditional leather material cutting robot visual recognition system cannot recognize leather thickness changes, material waste is easily caused by subcutaneous defects, natural curling deformation of leather is difficult to adapt, and the edge precision error is large are solved, the rejection rate can be further reduced through multispectral and 3D defect recognition, and the production efficiency is improved. Three-dimensional dynamic compensation enables edge errors to be further reduced, and real-time force control further reduces the occurrence rate of crush damage.

The invention discloses a robot vision external parameter calibration method based on blind deblurring and natural vibration measurement, and the method comprises the steps: collecting a continuous image sequence in a vibration environment through a high-frame-rate camera, extracting pixel-level motion information in combination with an advanced image deblurring deep network and an optical flow feature tracking technology, and carrying out the self-vibration measurement. And further restoring a three-dimensional displacement signal caused by natural vibration of the camera based on a pinhole imaging model, and obtaining ideal position information through frequency domain analysis to realize high-precision visual system external parameter calibration under a vibration condition.

The invention relates to the field of robot vision positioning, and discloses an intelligent hand-eye calibration and self-adaptive correction system and method. According to the method, a mechanical arm is controlled to drive a camera to collect multi-modal calibration data, a convolutional neural network is utilized to identify a calibration plate mark point, and a three-dimensional coordinate under a camera coordinate system is calculated in combination with depth information; meanwhile, on the basis of an encoder and torque data, flexible deformation of the mechanical arm is compensated through a kinematic model and a self-adaptive rigidity model, and three-dimensional coordinates under a base coordinate system are obtained. And obtaining a hand-eye transformation matrix by solving a transformation relation between the two point sets. And repeatedly calibrating before and after operation, inputting the difference of the two transformation matrixes into the fault diagnosis neural network, outputting fault type probability distribution, and generating a maintenance strategy. According to the invention, high-precision hand-eye calibration, automatic compensation of flexible deformation and intelligent diagnosis of system state change can be realized, so that the long-term precision and reliability of a visual positioning system are improved.

An AGV robot vision acquisition and avoidance control method based on deep thinking relates to the field of non-electrical variable control or adjustment systems, and comprises the following steps: acquiring real-time motion parameters of an automatic guided vehicle, calculating a key path point set and constructing feature vectors; collecting a visual image and depth data, extracting obstacle features through grid division, and generating a three-dimensional feature model; mapping the path points to a three-dimensional model to generate a dynamic scene sequence, and calculating a distance value with a dynamic target to determine a risk level; analyzing the scene by using a large language model, generating an obstacle avoidance scheme containing an obstacle avoidance track, speed and steering, and storing the obstacle avoidance scheme in a scheme library; current scene features are obtained, candidate schemes are selected through similarity matching, and the optimal scheme is selected for execution through safety scoring. By implementing the method, the obstacle avoidance success rate of the automated guided vehicle can be improved.

The invention provides a robot vision object semantic understanding and posture generation method based on a large model, and belongs to the technical field of image processing. Comprising the following steps: S1, inputting an image and a 3D model; s2, object detection; s3, multi-modal feature alignment is carried out; s4, diffusion model sampling; s5, performing geometric screening; s6, carrying out NeRF (New Random Field) morphological modeling; s7, optimizing the optical flow; s8, joint loss calculation; s9, confidence coefficient analysis; and S10, outputting the attitude junction. According to the invention, through an innovative single-view rendering-optical flow optimization closed loop strategy, the calculation overhead is significantly reduced and the estimation precision is improved. According to the method, the algorithm performance in a complex scene is remarkably improved through multi-scale feature fusion and a confidence decomposition strategy.

The invention provides a road foreign matter detection method based on tunnel inspection robot vision and related equipment, and relates to the technical field of intelligent traffic tunnel inspection robots. The method comprises the following steps: training by using an existing tunnel robot video data set to obtain an ROI region and a road object instance segmentation model; acquiring a video stream data frame of the inspection robot, generating an edge binary image through edge detection, obtaining related mask data through model reasoning, and recording classified foreign matter surrounding frame data; processing and combining various data, judging residual objects through morphological operation and contour detection, and determining unclassified foreign matter bounding box data; then reporting two types of foreign matter bounding box data according to rules; and finally, generating an annotation file according to the unclassified foreign matters, and updating the road object instance segmentation model. By implementing the method, tunnel road foreign matters can be timely and accurately detected, potential safety hazards are reduced, smooth tunnel traffic is guaranteed, and the intelligent and automatic level of tunnel inspection is improved.

The invention relates to the technical field of agricultural picking robot visual perception, and discloses a strawberry fruit detection and picking key point positioning method and model based on deep learning. The method comprises the following steps: acquiring strawberry RGB-D images to construct a data set; on the basis of a YOLOv8-Pose architecture, global feature extraction is enhanced by constructing a C2f-MAMBA module of a fusion state space model, an HWD module is introduced to retain detail features, and a Faster Net Block is used for weight reduction, so that an STRAW-MAMBA model is constructed; outputting coordinates of five key points such as a fruit bounding box and a shear point by the model, and performing geometric compensation on a shielding point; and finally, calculating three-dimensional space coordinates of the key points in combination with depth information. According to the method, a C2f-MAMBA module based on Mama is innovatively introduced, global feature association is effectively modeled, a multi-scale attention mechanism is combined, the defect that long-range dependence is difficult to capture in a traditional CNN is overcome, and the detection and positioning precision of strawberry fruits and picking key points thereof in a complex background is remarkably improved.

The invention provides a binocular vision global positioning system and method for a mobile robot, and relates to the technical field of robot vision navigation, and the method comprises the steps: carrying out the adaptive exposure compensation of binocular images of a left camera and a right camera, extracting edge contour feature points irrelevant to illumination, and generating a high-quality depth feature map through the combination of parallax calculation; meanwhile, feature units are constructed according to the inflection points, so that the global position of the mobile robot is determined. Therefore, the problem that the global positioning precision is reduced due to unstable visual feature extraction caused by rapid change of illumination intensity of the mobile robot in a multi-region switching scene can be solved to a certain extent.

Apparatuses, systems, and techniques are disclosed for controlling a robot to execute a task. In at least one embodiment, a current image of the robot in an environment and a text describing the task are obtained. A future image of the robot in the environment is predicted based on the current image and the text. Subsequently, one or more actions are predicted based on the current image, the future image, and the text. The one or more actions can move the robot from a first state corresponding to the current image to a second state corresponding to the future image. The robot executes the sequence of actions to move in the environment.

The invention discloses a distribution network hot-line work robot pose estimation system and method, and belongs to the technical field of robot visual perception. The system comprises an input preprocessing module which is used for carrying out noise reduction and enhancement processing on an RGB-D image; the shared feature extraction module is used for extracting multi-scale universal features based on a lightweight convolution architecture; the 6D pose estimation module is used for processing the image based on the neural implicit field to obtain a pose estimation result; and the joint optimization module is used for realizing detection and pose estimation shared feature extraction through cooperation of a multi-task loss function and a pose estimation result. The system adopts adaptive median filtering and homomorphic filtering to eliminate noise and uneven illumination, and bilateral filtering optimizes a depth map; a bottleneck structure and cavity convolution are introduced into feature extraction, and a rank enhancement linear attention module is embedded; according to the pose estimation, a geometric field and an appearance field are modeled through a neural implicit field, and pose hypotheses are generated and optimized. According to the method, the problems of low pose estimation precision and poor real-time performance in the distribution network live working environment are solved, and the working safety and efficiency of the robot are improved.

The invention is suitable for the technical field of artificial intelligence, and provides a robot visual navigation optimization system based on edge computing, the system comprises a visual sensor module, an edge computing unit, a navigation algorithm module, a communication module and a robot executor, the visual sensor module is used for collecting high-definition images or video streams, and the edge computing unit is used for processing the high-definition images or video streams; the edge calculation unit is responsible for locally executing a visual navigation algorithm, the navigation algorithm module is deployed on the edge calculation unit, and the navigation algorithm module is used for executing a visual odometer, synchronous positioning and map construction, environment perception, obstacle detection, local path planning and motion control; and the communication module is used for carrying out model updating, task scheduling and non-real-time data uploading with a cloud. Complex visual data processing and navigation decision are sunk to the robot body or near-end edge equipment, so that the navigation precision and the response speed of the robot in complex and diversified environments are remarkably improved, and the energy consumption efficiency is optimized.

The invention relates to a calibration method of a binocular camera and a laser radar based on a hollow ellipse calibrator, and belongs to the field of robot vision. Hamming code graticule marks are respectively arranged at four corners of one side face of a square aluminum plate of the hollow elliptical calibrator in the clockwise direction, an elliptical through hole is formed in the middle of the hollow elliptical calibrator, and the center of the elliptical through hole coincides with the center of the square aluminum plate; and the long axis of the elliptical through hole is positioned in the vertical direction. A hardware system of the calibration method comprises a binocular camera, an upper computer, a 16-line laser radar and a hollow ellipse calibrator. An image processing program, a three-dimensional point cloud processing program and a mathematical calculation program are arranged in the upper computer; the calibration method comprises the following operation steps: (1) carrying out image feature recognition on the hollow ellipse calibrator; (2) carrying out point cloud feature recognition on the hollow ellipse calibrator; (3) calculating external parameters of the camera and the 16-line laser radar; and (4) determining a calibration result. According to the calibration method, the calibration accuracy and robustness are improved, and the calibration method is suitable for practical application.

According to the robot vision calibration device and method, through collaborative design of the industrial personal computer, the vision system, the robot, the calibration jig and the calibration plate, the problems caused by complex structure, low calibration precision and poor adaptability in the prior art are solved. A floating mechanism of the calibration jig is combined with a displacement sensor to realize multi-degree-of-freedom dynamic displacement compensation, so that mechanical positioning errors are remarkably reduced; the conical surface guide structure and the reference hole layout of the calibration plate optimize the stability of the positioning reference, and ensure the accurate embedding of the calibration needle. The flexible installation of the visual system adapts to different scene requirements. According to the calibration method, the generation of a high-precision visual calibration matrix is realized through the processes of multi-reference hole coordinate recording, calibration ring coordinate calculation and visual image acquisition and matching. According to the whole scheme, through a dynamic compensation mechanism and intelligent process design, the calibration precision, efficiency and robustness are improved, meanwhile, the operation complexity is reduced, and reliable technical support is provided for an industrial automation scene.

The invention discloses a three-dimensional calibration automatic system suitable for a six-axis physiotherapy robot, and relates to the technical field of robot vision calibration and automatic control, and the system comprises an equipment communication management module which automatically identifies and distributes connection equipment through an intelligent detection technology, and outputs an equipment control instruction; the data acquisition and processing module is used for identifying 13-point feature pixels through RGB information, performing deep matching, calculating 13-point theoretical feature three-dimensional information and outputting an optimized calibration matrix MB2R in combination with a transformation relation between the camera and the tail end of the mechanical arm and a base coordinate transformation relation; and the result verification and backup uploading module is used for converting the relative position of the three-dimensional feature platform and the robot, acquiring three-dimensional information of an evaluation point, controlling the mechanical arm to move to an optimization point position, and calculating a pixel difference value and a height difference value. The method has the advantages that full-automatic millimeter-level calibration precision is achieved through a collaborative structure of multi-protocol heterogeneous equipment intelligent integration, iterative optimization algorithm fitting and closed-loop double-index verification.

The invention discloses a robot vision system construction and control method, and the method comprises the following steps: collecting environment image data and robot motion state data, carrying out the real-time preprocessing of the environment image data, generating standardized image data, and carrying out the real-time monitoring of the environment through setting a dynamic environment sensing module. When real-time grabbing operation guided by robot vision is carried out, an environment light interference real-time judgment mechanism is established, differential compensation strategies are set for different working condition scenes, the stability of visual data collection under the complex illumination condition is guaranteed, and meanwhile compensated light parameters are synchronized to an image processing unit; the problem of feature recognition distortion caused by abnormal environmental reflection can be reduced in real time, the accuracy of robot target positioning is ensured, the track error of visual guidance is reduced, and the completeness of a high-precision assembly task is ensured when it is detected that the in-place posture offset exceeds a safety threshold value.

The invention discloses a video tracking control method and device for a non-linear moving target based on adaptive sliding mode control, which are suitable for accurate tracking of a dynamic target. The system is composed of a visual sensing module, a main control module and an execution mechanism, the visual sensing module collects position information of a target object in real time and transmits the position information to a main control chip through serial port communication, the main control chip calculates the error between the target position and the current position of the system and the error change rate, and a sliding mode surface is designed based on sliding mode control. Meanwhile, sliding mode control parameters are dynamically adjusted in combination with a self-adaptive control algorithm, so that the robustness of the system to uncertainty and external interference is enhanced, and meanwhile the chattering phenomenon is restrained. A stepping motor control signal is optimized by introducing a dynamic acceleration and deceleration algorithm, so that control smoothness and response precision are ensured, and real-time accurate tracking of a target object is realized. The method has strong robustness, high dynamic performance and control precision on the premise of not depending on a target kinematic model, can be widely applied to the fields of unmanned aerial vehicle tracking, intelligent monitoring, robot vision and the like, and effectively solves the problems of easy buffeting, response hysteresis, insufficient control precision and the like during nonlinear target tracking in the prior art.

The application discloses a mechanical arm cable assembly method and system, and relates to the technical field of robot vision perception and control, and comprises the following steps: acquiring point cloud data of a working environment; fitting a cable center line according to the point cloud data and determining a relative pose matrix of a cable terminal and a target slot; determining an initial alignment attitude matrix of a mechanical arm end according to the relative pose matrix, so as to obtain initial joint angles of the mechanical arm and use the initial joint angles as a starting point of an assembly action; after selecting a grabbing point on the cable center line and planning an insertion path, entering an insertion stage from the starting point; according to detection of an insertion force and an insertion direction, performing error calculation and compensation until the insertion force change rate and the end pose error meet a set condition, and then determining that the assembly is completed. The application realizes self-perception, self-judgment and self-adjustment flexible assembly, improves assembly precision and robustness, and is suitable for automatic grabbing, guiding and insertion operation of flexible cables in industrial automation production.

The invention provides a robot visual language navigation method, device and equipment based on key point guidance. The method comprises the following steps: extracting a keyword of a voice instruction as a target text; based on a multi-modal pre-training knowledge base, determining knowledge prototype features corresponding to the target text, and determining a target similarity with the maximum similarity from the similarity between the knowledge prototype features and a plurality of regional features corresponding to the RGB image; and under the condition that the target similarity is greater than a preset threshold value, determining a target object in the region feature in the matching pair corresponding to the target similarity, determining a target coordinate of the target object based on the depth information of the target object, and determining a navigation path of the robot based on the target coordinate and a robot moving model. According to the method, through vision-language cross-modal matching, the similarity between a target text and an image region is calculated to realize navigation supervision; efficient fusion of multi-modal information can be realized in a complex scene, and the precision and robustness of robot navigation are remarkably improved.