687 results about "Robot vision" patented technology

The invention belongs to the technical field of image processing, and particularly relates to an object six-degree-of-freedom pose estimation method based on three-dimensional geometric information registration, which comprises a target detection module, a three-dimensional point cloud generation model and a geometric information registration pose estimation module. According to the method, geometric reconstruction, a geometric information high-precision registration technology and an object pose estimation task are combined, a geometric reconstruction module introduces object geometric feature extraction and feature decoding to realize deep understanding and modeling of an algorithm on object geometric features, the whole process accords with geometric disorder, and the accuracy of object pose estimation is improved. And the accuracy and scene adaptability of the algorithm are further improved. According to the method, accurate estimation of the six-degree-of-freedom pose of the object can be achieved, and powerful support is provided for the fields of industrial automation, robot vision, AR / VR, automatic driving, medical robots and the like.

The invention discloses an AI-based neurosurgery auxiliary robot visual positioning system, and relates to the field of visual positioning, which comprises the steps of deploying and initializing structured light scanning equipment and near-infrared imaging equipment, carrying out multi-modal image acquisition on a surgical area, and carrying out standardization processing, space-time alignment and fusion on the acquired image. A multi-modal visual acquisition and preprocessing module, a three-dimensional tissue model construction and registration module, a visual-anatomical feature recognition and extraction module, an AI auxiliary positioning and path optimization module, an intraoperative dynamic perception and feedback control module and a target position confirmation and instruction output module are constructed. According to the method, high-precision identification and dynamic modeling of a brain tissue structure are realized, real-time identification, path planning and position correction can be carried out on a key anatomical structure in an operation process, the precision, the intelligent level and the intra-operation response capability of neurosurgery operation are remarkably improved, the operation risk is effectively reduced, and the positioning reliability and the automatic control efficiency are improved.

The invention discloses a robot visual identification, positioning and grabbing method based on RGBD point cloud, and the method comprises the following steps: 1, carrying out the camera point cloud data processing, synthesizing an RGB image and a depth image into a point cloud image, and carrying out the filtering and denoising processing of the point cloud data; step 2, target detection and capture attitude generation: performing target detection and instance segmentation on the RGB image, identifying a target object and generating a corresponding mask frame, and applying a mask to the depth image to extract depth point cloud data corresponding to the target object; obtaining the depth point cloud data, generating a plurality of groups of grabbing postures, and screening out an optimal grabbing posture; and thirdly, the mechanical arm grabbing execution module receives the optimal grabbing posture information output by the target detection and grabbing posture generation module and controls the robot to complete the grabbing action.

The invention relates to the field of image processing, and particularly discloses a welded pipe surface defect detection method based on robot visual inspection. The method comprises the steps that a robot carries a binocular camera and an annular LED light source and moves at a constant speed in the axial direction of a welded pipe to collect orthographic and inclined views, and a three-dimensional point cloud is constructed; and establishing a parameterized mapping function based on the point cloud, and converting the 3D coordinate into a 2D expansion surface coordinate. In the convolutional neural network, a first layer is inserted into a spatial transformation network to correct distortion of the expanded image, deformable convolution is adopted to extract edge, local deformation and specific defect response features, and standard convolution is combined to extract global features; and fusing multi-scale features and adding an attention mechanism to improve the weight of a defect region, and outputting a defect category and a bounding box offset after generating a candidate box. The method effectively solves the problems of stretching, deformation and defect distortion of welded pipe curved surface imaging, reduces the imaging difference of the same defect, and remarkably improves the defect positioning precision and recognition accuracy.

The invention relates to the technical field of welding automation and robot visual perception, and discloses a robot intelligent welding method and system.The method comprises the steps that a three-dimensional point cloud image is constructed through a laser triangulation method; filtering and enhancing the point cloud image, and extracting a weld point cloud set by adopting a U-Net semantic segmentation model; carrying out space path identification and attitude regression based on the set to obtain a track element of a position-attitude pair; fitting a path by adopting a cubic B-spline curve and interpolating to generate a continuous welding track; and in the welding process, visual feedback is combined, and proportional differential control logic is introduced for closed-loop track error compensation. Compared with the technical problem that in the prior art, weld joint path recognition and accurate tracking cannot be stably achieved under complex structures such as curved surfaces and variable cross sections, a perception control closed loop based on visual servo is constructed, high-precision automatic welding of the complex weld joint structures is achieved, and the track fitting precision is improved.

The invention discloses a robot vision-inertia SLAM method and device and a medium, and belongs to the technical field of computer vision and robot navigation. The method comprises the following steps: synchronously acquiring images and inertial data through a robot binocular camera and an IMU, performing feature enhancement on an original image in combination with a pre-trained deep learning model, introducing an adaptive brightness compensation mechanism and designing an image light supplementing module based on a generative adversarial network (GAN), recovering low-illumination image details, and improving the image quality of a low-illumination area; an entropy-based adaptive dynamic interference rejection algorithm is provided, and dynamic interference feature points are rejected in combination with IMU (Inertial Measurement Unit) data; a low-rank approximate improved graph optimization algorithm is adopted, and global map construction and pose optimization are accelerated; and through entropy-based nonlinear dynamic smoothing coefficient adjustment, the track stability is improved. According to the method, the positioning precision and robustness in a low-light environment are improved, the calculation efficiency is improved, and dynamic interference is effectively resisted.

The invention discloses an automatic dovetail welding seam grinding control method based on robot visual positioning, and relates to the technical field of visual positioning. The method comprises the steps that after welding is completed, a dovetail welding seam image is collected and converted into a three-dimensional coordinate through a vision algorithm, and a three-dimensional point cloud is generated; smooth interpolation is performed on the three-dimensional point cloud through a B spline curve method to obtain a parameterized curve, and a control point set is optimized through a genetic algorithm to generate a global optimal polishing path; the polishing robot executes a task according to a path, a tail end sensor collects a real-time path and calculates a deviation value with a global path, and when the deviation value exceeds a preset threshold value, inverse kinematics is triggered to solve and correct the path; the hardness of the dovetail weld is measured through laser-induced breakdown spectroscopy, a comprehensive hardness value is obtained in combination with a matrix hardness database, meanwhile, a visual sensor collects the surface state, and polishing process parameters are dynamically adjusted according to the surface state; after the task is completed, the welding seam angle and flatness are detected. According to the method, the optimal path is generated through visual positioning, and automatic grinding of the dovetail welding seam is achieved.

The invention relates to the technical field of robot visual perception and motion control, and discloses a motion control method and device based on physical reference, equipment and a medium, and the method comprises the steps: obtaining instruction information, a multi-view image and movable assembly pose information; processing the multi-view image according to the instruction information to generate target segmentation information; generating a scale normalization point cloud and a model estimation baseline; determining a physical reference baseline and generating a scale calibration factor; converting the scale normalization point cloud into a physical space point cloud by using a scale calibration factor; extracting a three-dimensional relative position of the target object relative to the movable component in combination with the target segmentation information; an action instruction is generated based on the multi-modal input. According to the method, physical scale alignment of the point cloud is realized through physical reference baseline calibration, so that a visual reconstruction result has real space significance, an accurate action instruction is generated, and the robot space understanding and operation precision is improved.

The invention proposes a method and system for automatically identifying and controlling a robot to enter the undercarriage of roll carts. Specifically, the method and system proposed in the invention belong to the field of autonomous mobile robots (AMRs) applied in the transportation of goods in warehouses. The invention relates to the field of robot vision and automatic control.

The invention discloses a joint denoising method for robot visual motion prediction, and the method comprises the steps: constructing a unified generative model through fusing an image and a depth map collected by a depth camera, motion data collected by CAN line communication of a Piper mechanical arm, and a tactile image collected by a Gelsight Mini tactile sensor; the method comprises two steps of data acquisition and input coding, and joint denoising and generation: firstly, multi-modal data are coded into low-dimensional potential representation, and then future images, depth maps, tactile data and robot actions are cooperatively predicted through a joint denoising framework based on Transform. A mask self-attention mechanism is innovatively introduced, information interaction between modes is dynamically adjusted, action generation is guided through tactile feedback, and the force control precision is improved. The model adopts a de-noising diffusion probability loss function to jointly optimize multi-modal prediction, so that the output consistency is ensured. According to the method, the robustness and the accuracy of flexible operation of the robot are remarkably improved.

The invention discloses a robot visual identification decision control method based on deep learning, and relates to the technical field of control systems. Comprising the steps of constructing a thinking decision tree according to an input instruction and a real-time visual image, and obtaining a linear relationship between a middle-layer leaf node as a main body and a top-layer leaf node for realizing association between a current visual feature and a task intention. According to the method, deep fusion and unified representation of multi-source heterogeneous data are realized by constructing a multi-layer thinking decision tree structure and combining multi-modal feature fusion and a deep learning engine, and text instructions, visual images and other different modal information are effectively integrated by constructing a fusion search engine and a semantic alignment mechanism, so that the multi-source heterogeneous data fusion and unified representation are realized. Cooperative processing of accurate analysis of task intentions and environment perception is realized, and real-time response and decision accuracy of the service robot to complex tasks are improved.

The invention discloses a digital twin interaction control system based on robot vision and tactile fusion, and the system comprises a data collection module which is used for collecting visual image data and tactile sensing data; the preprocessing module is used for preprocessing; the multi-mode perception modeling module is used for executing spatial perception modeling and tactile state modeling; the fusion state vector construction module is used for generating a fusion state vector; the digital twinborn mapping module is used for constructing a digital twinborn body; the simulation interaction control module is used for executing interactive action control and generating a control instruction parameter group; the action execution and feedback module is used for controlling the robot to execute the actual action and updating the fusion state vector; and the dynamic evaluation and self-adaptive adjustment module is used for dynamically optimizing and adjusting the fusion state vector. Visual and touch information of the robot is fused, a digital twinning synchronous control mechanism is constructed, and the interactive optimization method which is fast in operation response, high in execution precision and stable in control process is achieved.

The invention discloses a visual tracking and real-time guiding robot and conveying belt following method, and belongs to the technical field of book packaging, the visual tracking and real-time guiding robot and conveying belt following method comprises a conveying belt device used for conveying articles and having adjustable running speed and direction control functions; the robot main body is arranged on one side of the conveying belt device; the fixed vision device is connected with the robot main body; through cooperative work of the robot vision module and the independent fixed vision device on the robot body, all-directional vision coverage of the conveying belt and the surrounding environment is achieved, and the robot vision module focuses on local detail information of the robot and objects on the conveying belt; the relative position and posture change of the robot and an object can be captured in a close-range and high-precision mode in the operation process of the robot, and the fixed vision device continuously monitors the overall condition of the conveying belt from the overall perspective.

The invention relates to the technical field of robot visual guidance, in particular to an out-of-order target recognition and positioning method based on binocular laser three-dimensional scanning imaging, which comprises the following steps of: performing laser scanning on an object in a target area, and synchronously acquiring laser stripe images by adopting a binocular camera to obtain a target area image sequence; extracting a laser stripe center line of the target area image sequence to obtain auxiliary features; performing three-dimensional point cloud reconstruction on the auxiliary features to obtain original point cloud data; compensating the original point cloud data to obtain fused point cloud data; on the basis of a spherical multi-view sampling and positioning algorithm framework, off-line model point cloud features are constructed, and object recognition and pose matching are carried out on fused point cloud data; and visualizing an object identification and pose matching result. The method has the characteristics of stable identification and positioning results, high applicability in complex scenes, high identification accuracy and the like.

The invention relates to the technical field of tumor interventional operations, and discloses a visual navigation method based on a tumor interventional operation robot. The method comprises the following steps: acquiring real-time medical image data of a tumor area containing multi-modal imaging information so as to comprehensively present anatomical details; and performing three-dimensional reconstruction on the image data to generate a tumor area three-dimensional anatomical structure model capable of visually displaying a space structure. Key anatomical feature points are extracted based on the model, space coordinates are calculated, a surgical robot intervention path is planned according to the coordinates, and an initial navigation track is generated; and continuously collecting real-time pose data of the robot in an operation, dynamically matching the real-time pose data with the initial navigation trajectory, adjusting motion parameters according to a matching result, and generating a corrected navigation instruction. The method can reflect the intraoperative anatomy condition in real time, dynamically optimize the path, solve the problems that traditional navigation depends on preoperative static images and lacks real-time adjustment, reduce operative complications and improve the treatment effect of patients.

The invention belongs to a robot navigation method, and aims to solve the technical problems of difficulty in establishing a geometric map in a non-structural scene, dependence on tags from laser radar data, easiness in ignoring long-distance features and slow reasoning time in the conventional navigation or exploration method. According to the robot visual navigation method based on comparative learning and the related device, after current observation is obtained, the current observation is input into a trained observation encoder to obtain a corresponding feature vector, and then a pre-selected batch action sequence passes through an action encoder to extract the corresponding feature vector; and calculating the similarity between the current observation and each action sequence through a vector inner product, selecting the action sequence with the highest similarity, and inputting the selected action sequence into a robot controller for controlling the robot to execute corresponding actions. Compared with a method for directly generating the action sequence, the method has the advantages that the difficulty of selecting one action sequence is lower, and the expansibility and generalization are better.

The invention relates to a leather material cutting robot and a cutting method thereof, and belongs to the technical field of material cutting, the leather material cutting robot comprises a human-computer interaction interface, a CAD / CAM system, a multi-mode visual perception unit, a central processing system, a dynamic compensation cutting unit and a quality closed-loop system, and the multi-mode visual perception unit comprises a high-resolution multispectral imaging module and a 3D scanning module. According to the method, the problems that a traditional leather material cutting robot visual recognition system cannot recognize leather thickness changes, material waste is easily caused by subcutaneous defects, natural curling deformation of leather is difficult to adapt, and the edge precision error is large are solved, the rejection rate can be further reduced through multispectral and 3D defect recognition, and the production efficiency is improved. Three-dimensional dynamic compensation enables edge errors to be further reduced, and real-time force control further reduces the occurrence rate of crush damage.

The invention discloses a robot vision external parameter calibration method based on blind deblurring and natural vibration measurement, and the method comprises the steps: collecting a continuous image sequence in a vibration environment through a high-frame-rate camera, extracting pixel-level motion information in combination with an advanced image deblurring deep network and an optical flow feature tracking technology, and carrying out the self-vibration measurement. And further restoring a three-dimensional displacement signal caused by natural vibration of the camera based on a pinhole imaging model, and obtaining ideal position information through frequency domain analysis to realize high-precision visual system external parameter calibration under a vibration condition.

The invention relates to the field of robot vision positioning, and discloses an intelligent hand-eye calibration and self-adaptive correction system and method. According to the method, a mechanical arm is controlled to drive a camera to collect multi-modal calibration data, a convolutional neural network is utilized to identify a calibration plate mark point, and a three-dimensional coordinate under a camera coordinate system is calculated in combination with depth information; meanwhile, on the basis of an encoder and torque data, flexible deformation of the mechanical arm is compensated through a kinematic model and a self-adaptive rigidity model, and three-dimensional coordinates under a base coordinate system are obtained. And obtaining a hand-eye transformation matrix by solving a transformation relation between the two point sets. And repeatedly calibrating before and after operation, inputting the difference of the two transformation matrixes into the fault diagnosis neural network, outputting fault type probability distribution, and generating a maintenance strategy. According to the invention, high-precision hand-eye calibration, automatic compensation of flexible deformation and intelligent diagnosis of system state change can be realized, so that the long-term precision and reliability of a visual positioning system are improved.

The invention discloses an underwater robot visual image compression, decompression and transmission method and device. The method comprises the steps of constructing an image compression model used for underwater robot visual image compression, training the image compression model, constructing a visual image coding end based on the trained image compression model, performing compression coding on the underwater robot visual image by using the visual image coding end, generating code stream data, and outputting the code stream data. And constructing a visual image compression end based on the trained image compression model so as to complete compression of the visual image of the underwater robot, and constructing a visual image decoding end based on the trained image compression model, the visual image decoding end being used for decoding the code stream data and reconstructing the image so as to generate a reconstructed visual image of the underwater robot. The underwater robot visual image compression and decompression method provided by the invention is high in compression ratio and good in image reconstruction performance, and can be applied to compression and transmission tasks of underwater robot visual images.

An AGV robot vision acquisition and avoidance control method based on deep thinking relates to the field of non-electrical variable control or adjustment systems, and comprises the following steps: acquiring real-time motion parameters of an automatic guided vehicle, calculating a key path point set and constructing feature vectors; collecting a visual image and depth data, extracting obstacle features through grid division, and generating a three-dimensional feature model; mapping the path points to a three-dimensional model to generate a dynamic scene sequence, and calculating a distance value with a dynamic target to determine a risk level; analyzing the scene by using a large language model, generating an obstacle avoidance scheme containing an obstacle avoidance track, speed and steering, and storing the obstacle avoidance scheme in a scheme library; current scene features are obtained, candidate schemes are selected through similarity matching, and the optimal scheme is selected for execution through safety scoring. By implementing the method, the obstacle avoidance success rate of the automated guided vehicle can be improved.

The invention provides a robot vision object semantic understanding and posture generation method based on a large model, and belongs to the technical field of image processing. Comprising the following steps: S1, inputting an image and a 3D model; s2, object detection; s3, multi-modal feature alignment is carried out; s4, diffusion model sampling; s5, performing geometric screening; s6, carrying out NeRF (New Random Field) morphological modeling; s7, optimizing the optical flow; s8, joint loss calculation; s9, confidence coefficient analysis; and S10, outputting the attitude junction. According to the invention, through an innovative single-view rendering-optical flow optimization closed loop strategy, the calculation overhead is significantly reduced and the estimation precision is improved. According to the method, the algorithm performance in a complex scene is remarkably improved through multi-scale feature fusion and a confidence decomposition strategy.

The invention discloses a zero sample object instance segmentation system and method in a robot environment, and belongs to the technical field of robot visual perception, and the method comprises the steps: firstly carrying out the Vibriis color mapping preprocessing of a depth image, and then generating an initial object mask candidate through an SAM (Segment Anything Model) model; meanwhile, a pre-trained Vision Transform is used as a feature description model of a scene to process the image, an attention map of the last layer is extracted, and a feature weighting mechanism is constructed based on the information entropy of the attention map; a similarity matrix of the image blocks and the background blocks is calculated to remove non-object masks, a K-Medoids clustering algorithm is adopted for each remaining object mask candidate to obtain representative sampling points, and finally the points serve as prompt information to be input into an SAM model to obtain an accurate object instance segmentation result. According to the method, the zero sample generalization ability of the visual basic model is fully utilized, accurate segmentation of the unseen object can be realized without additional training, and the method has good universality and practical value.

The invention belongs to the technical field related to robot visual control, and discloses a method and system for cooperatively controlling a mechanical arm to move based on visual servo and force sense admittance. The method comprises the steps that a mechanical arm moves to an assembly target position, and an image collection device collects an image of a target point at the assembly target position as an expected image; the tail end of the mechanical arm clamps the to-be-assembled workpiece to move to an assembly target position, an image collection device collects an image of a target point in real time in the moving process of the tail end of the mechanical arm, and the image collected in real time is compared with an expected image so as to calculate the movement speed vc of the tail end of the mechanical arm; when the to-be-assembled workpiece is in contact with the pin at the assembly target position, the force sensor measures the force and the torque during contact, and the movement speed vf of the tail end of the mechanical arm is calculated through the force and the torque; and calculating the movement speed of the tail end of the mechanical arm at the next moment by utilizing the movement speeds vc and vf of the mechanical arm. According to the invention, the problem that visual control and force control cannot be cooperatively controlled in the prior art is solved.

The invention provides a road foreign matter detection method based on tunnel inspection robot vision and related equipment, and relates to the technical field of intelligent traffic tunnel inspection robots. The method comprises the following steps: training by using an existing tunnel robot video data set to obtain an ROI region and a road object instance segmentation model; acquiring a video stream data frame of the inspection robot, generating an edge binary image through edge detection, obtaining related mask data through model reasoning, and recording classified foreign matter surrounding frame data; processing and combining various data, judging residual objects through morphological operation and contour detection, and determining unclassified foreign matter bounding box data; then reporting two types of foreign matter bounding box data according to rules; and finally, generating an annotation file according to the unclassified foreign matters, and updating the road object instance segmentation model. By implementing the method, tunnel road foreign matters can be timely and accurately detected, potential safety hazards are reduced, smooth tunnel traffic is guaranteed, and the intelligent and automatic level of tunnel inspection is improved.

The invention relates to the technical field of agricultural picking robot visual perception, and discloses a strawberry fruit detection and picking key point positioning method and model based on deep learning. The method comprises the following steps: acquiring strawberry RGB-D images to construct a data set; on the basis of a YOLOv8-Pose architecture, global feature extraction is enhanced by constructing a C2f-MAMBA module of a fusion state space model, an HWD module is introduced to retain detail features, and a Faster Net Block is used for weight reduction, so that an STRAW-MAMBA model is constructed; outputting coordinates of five key points such as a fruit bounding box and a shear point by the model, and performing geometric compensation on a shielding point; and finally, calculating three-dimensional space coordinates of the key points in combination with depth information. According to the method, a C2f-MAMBA module based on Mama is innovatively introduced, global feature association is effectively modeled, a multi-scale attention mechanism is combined, the defect that long-range dependence is difficult to capture in a traditional CNN is overcome, and the detection and positioning precision of strawberry fruits and picking key points thereof in a complex background is remarkably improved.

The invention provides a binocular vision global positioning system and method for a mobile robot, and relates to the technical field of robot vision navigation, and the method comprises the steps: carrying out the adaptive exposure compensation of binocular images of a left camera and a right camera, extracting edge contour feature points irrelevant to illumination, and generating a high-quality depth feature map through the combination of parallax calculation; meanwhile, feature units are constructed according to the inflection points, so that the global position of the mobile robot is determined. Therefore, the problem that the global positioning precision is reduced due to unstable visual feature extraction caused by rapid change of illumination intensity of the mobile robot in a multi-region switching scene can be solved to a certain extent.

Apparatuses, systems, and techniques are disclosed for controlling a robot to execute a task. In at least one embodiment, a current image of the robot in an environment and a text describing the task are obtained. A future image of the robot in the environment is predicted based on the current image and the text. Subsequently, one or more actions are predicted based on the current image, the future image, and the text. The one or more actions can move the robot from a first state corresponding to the current image to a second state corresponding to the future image. The robot executes the sequence of actions to move in the environment.

The invention relates to the technical field of visual identification and detection of polishing robots, in particular to a multi-type polishing area detection algorithm for casting autonomous polishing. Comprising the following steps of S1, process type classification, S2, defect type classification, S3, bulge type defect detection, S4, crator defect detection, S5, rough surface defect detection and S6, point-by-point grinding process detection. According to a grinding target area detection algorithm provided by the invention, a position sensing basis is provided for grinding track planning by classifying grinding areas; different detection algorithm libraries are formed and deployed in a system of the grinding robot, the adaptability to detection of the to-be-ground area can be enhanced, and the working capacity of the grinding robot is improved.

The invention discloses a distribution network hot-line work robot pose estimation system and method, and belongs to the technical field of robot visual perception. The system comprises an input preprocessing module which is used for carrying out noise reduction and enhancement processing on an RGB-D image; the shared feature extraction module is used for extracting multi-scale universal features based on a lightweight convolution architecture; the 6D pose estimation module is used for processing the image based on the neural implicit field to obtain a pose estimation result; and the joint optimization module is used for realizing detection and pose estimation shared feature extraction through cooperation of a multi-task loss function and a pose estimation result. The system adopts adaptive median filtering and homomorphic filtering to eliminate noise and uneven illumination, and bilateral filtering optimizes a depth map; a bottleneck structure and cavity convolution are introduced into feature extraction, and a rank enhancement linear attention module is embedded; according to the pose estimation, a geometric field and an appearance field are modeled through a neural implicit field, and pose hypotheses are generated and optimized. According to the method, the problems of low pose estimation precision and poor real-time performance in the distribution network live working environment are solved, and the working safety and efficiency of the robot are improved.