2466 results about "Depth map" patented technology

The invention discloses a three-dimensional scene reconstruction method and device based on large model geometric prior, and a medium, and aims to solve the problems that a conventional 3DGS is liable to have artifacts and detail loss in geometric discontinuity, data redundancy and illumination variation scenes, and predicts a dense depth map and a normal map from a monocular image by using a pre-trained large model. The position and form of the Gaussian kernel are constrained as additional geometric priori; a primitive adjustment strategy based on kernel density estimation is introduced in the training stage, small Gaussian primitives with similar structures and adjacent spaces are combined into a large Gaussian primitive, the rendering quality is kept, redundancy is reduced, and the volume of the model is reduced; an exposure coefficient is adaptively estimated for each input image, an exposure compensation image loss function is constructed, and floating artifacts caused by illumination differences at shooting moments are eliminated. Experiments show that compared with the prior art, the method improves the three-dimensional reconstruction precision and real-time rendering quality of complex illumination and less-texture areas in a public data set and an unmanned aerial vehicle aerial photography scene.

Disclosed in the present invention is a fruit stem pose recognition method used for a pomelo fruit picking apparatus, the method comprising the following steps: preprocessing image data of a scene to be recognized, so as to obtain preprocessed image data, the image data comprising an RGB map and a depth map; inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth map of a single pomelo fruit; on the basis of the depth map of a single pomelo fruit, recognizing an elliptical fruit pose feature region, and further using the Hough ellipse detection algorithm to obtain an ellipse tilt angle of the fruit pose feature region; and, by means of the ellipse tilt angle of the fruit pose feature region, calculating and obtaining the position and orientation of a pomelo fruit stem. The method integrates a plurality of advanced technologies to achieve optimization of a whole process from image data preprocessing to fruit stem pose accurate recognition, thus improving the recognition accuracy and working efficiency of automated pomelo fruit picking apparatuses, and reducing the mis-picking rate and the fruit damage rate.

The invention relates to the technical field of constructional engineering, and discloses a concrete crack intelligent identification and analysis platform based on image and point cloud fusion, the platform operates a concrete crack intelligent identification and analysis method, and the method comprises the following steps: S1, synchronously collecting image data and point cloud data of a concrete structure in the same scene; s2, establishing a unified world coordinate system and generating a depth map corresponding to the image; s3, generating image domain crack candidates; s4, generating a depth domain crack candidate; s5, performing weighted fusion on the image domain crack candidate and the depth domain crack candidate to generate a fusion crack response; s6, extracting a crack skeleton; s7, obtaining a crack three-dimensional model; and S8, selecting an optimal view angle to trigger re-acquisition of the crack area. Through a closed-loop feedback mechanism, an optimal view angle is selected for re-acquisition by calculating a comprehensive utility value after preliminary acquisition, so that information insufficiency caused by illumination, angle or sparse data is effectively made up.

The invention discloses an automobile part quality detection method and system based on artificial intelligence visual inspection, and belongs to the field of artificial intelligence machine visual inspection, and the method comprises the steps: firstly, carrying out the registration of a collected RGB image and a depth image, and extracting a part region through a saliency detection network; two-dimensional key points are extracted based on an RGB region image and are matched with key points of a three-dimensional model, an initial three-dimensional attitude is obtained by adopting a PnP algorithm, iterative registration is performed with the three-dimensional model in combination with a point cloud generated by a depth image, and a fine three-dimensional attitude is obtained. And calculating a geometric transformation matrix from the part to a standard front view attitude according to the attitude, and performing attitude correction on the RGB and depth region image. And then matching the corrected image with a standard template image by using a feature detection and matching network so as to correct the position of the detection window. And finally, the three-dimensional size of the part is calculated in the corrected detection window in combination with the depth value, and tolerance judgment is carried out. And the precision, the robustness and the automation level of online detection of the automobile parts can be obviously improved.

The invention relates to the technical field of virtual reality, in particular to an indoor structure reconstruction method based on a panoramic image scene understanding algorithm, and the method comprises the steps: S11, obtaining a plurality of equidistant columnar projection panoramic images of a current indoor scene through a panoramic camera or a panoramic image splicing algorithm; s12, performing semantic segmentation on the collected panoramic image by using SAM, deducing indoor ceiling, floor and wall areas of the panoramic image on the basis of a semantic segmentation result, and generating a multi-channel semantic graph; and S13, inputting the acquired RGB panoramic images and the multi-channel semantic map into a pre-trained panoramic image depth estimation model to obtain a depth map corresponding to each panoramic image. The method is used for automatically generating an indoor structure model, the scheme comprises the core steps of semantic segmentation and layout reasoning, depth estimation and point cloud construction, structure optimization and indoor model reconstruction and the like, and a complete indoor panoramic image scene understanding scheme is formed.

The invention discloses an electronic cabin assembly quality detection method based on three-dimensional vision. The method comprises the following steps: firstly, generating data for comparison during detection for a standard electronic cabin; shooting all to-be-detected objects in the to-be-detected electronic cabin to obtain a depth image and a color image of each to-be-detected object; according to the depth image and the color image of the to-be-detected object, combining with the data of the standard electronic cabin for comparison during detection; and whether screws of the electronic cabin to be detected are neglected and not installed in place, whether cable plugs are neglected and not installed in place, whether connectors and cable plugs in the wiring unit are wrongly matched, and whether the bending radius of cables is too small are detected in sequence. Through fusion of depth information, three-dimensional quantitative detection of the assembly quality is realized, and the detection precision, reliability and automation level are significantly improved.

The invention discloses an uncertainty mapping method and device based on selective learnable depreciation, and the method comprises the steps: S1, collecting color images and depth maps at different time points, and projecting the color images and the depth maps to a three-dimensional coordinate system; s2, predicting an evidence vector for each pixel by using an evidence generation network; s3, calculating total evidence intensity based on the evidence vector, and defining credibility and uncertainty based on the total evidence intensity; s4, constructing a noise mask to represent the observation quality, and predicting a damage coefficient by using a selective damage network; s5, the evidence intensity is adjusted in a zooming and depreciation mode; and S6, outputting a final probability, carrying out basic probability distribution of multi-frame fusion observation on the three-dimensional voxel grid, and constructing a semantic map containing uncertainty estimation. According to the method, the conflict rate and uncertainty in the semantic map fusion process can be effectively reduced, and the accuracy and credibility of the map are remarkably improved.

The invention discloses a depth-guided three-dimensional Gaussian reconstruction method and system suitable for a sparse view angle image, and belongs to the technical field of computer vision and three-dimensional reconstruction, and the method comprises the steps: carrying out the wavelet transformation super-resolution processing of a sparse multi-view angle image; outputting depth prior through a pre-trained monocular depth model, and extracting multi-view image features; constructing cross-view depth candidates by adopting a planar scanning stereo method, and generating initial depth distribution through feature similarity calculation; a self-attention-cross attention structure and deformable sampling are adopted to realize coarse-to-fine depth matching optimization; using an improved UNet network to fuse multi-scale features, and optimizing a depth estimation result; predicting three-dimensional Gaussian primitive parameters; and constructing a Gaussian field to generate a new visual angle image. According to the method, the problems of low accuracy, integrity and efficiency of existing sparse view angle three-dimensional reconstruction are solved. According to the invention, the precision and the detail fidelity of the depth map are improved, and high-fidelity three-dimensional reconstruction under the sparse visual angle condition is realized.

The invention discloses an indoor scene three-dimensional reconstruction method based on deep fusion and confidence modeling, and belongs to the technical field of computer vision. According to the method, composite data frames such as a color image, a depth map, an IMU (Inertial Measurement Unit) and a camera attitude are comprehensively utilized to carry out regional three-dimensional reconstruction on an indoor scene: firstly, the scene is divided into a smooth region (such as a wall, a ground, a ceiling, a glass plane, a mirror surface, a blackboard and other planes) and a complex curved surface region; aiming at the smooth area, adopting geometric prior guide plane fitting provided by a visual large model, and combining sensor attitude information to quickly reconstruct a regular plane model; for a complex curved surface area, a multi-frame point cloud fusion strategy is adopted to accumulate different view angle information, and a deep residual error refining network is utilized to recover curved surface details, so that a high-precision curved surface model is obtained. According to the method, the three-dimensional structure of the indoor scene can be efficiently reconstructed, the global framework of the smooth area is reserved, and the details of the surface of a complex object are depicted in detail.

The invention belongs to the technical field of computer vision three-dimensional scene repair, and provides an automatic driving three-dimensional scene repair method based on color point cloud prior guidance, which comprises the following steps: acquiring multi-source heterogeneous data; constructing an incomplete semantic two-dimensional Gaussian field according to the multi-source heterogeneous data, and generating an incomplete image sequence, an incomplete depth image sequence and an opacity image sequence; obtaining a color point cloud based on the instance segmentation image, the opacity image, the incomplete image and the incomplete depth image of the reference image; generating a texture pseudo view angle sequence based on the color point cloud; generating a repaired image sequence by taking the texture pseudo view angle sequence as a condition signal of the fine-tuning video diffusion model; generating a repair depth sequence based on the color point cloud; calculating a repair Gaussian loss function based on the repair image sequence and the repair depth sequence, and performing iterative repair optimization on the incomplete semantic two-dimensional Gaussian field to obtain a complete repair two-dimensional Gaussian field; the invention further discloses a computer device. And texture-geometry collaborative automatic driving scene three-dimensional repair is realized.

The invention discloses a three-dimensional scene reconstruction method and system based on monocular depth estimation, and belongs to the technical field of computer vision and three-dimensional reconstruction, and the method comprises the steps: carrying out the multi-scale feature coding of a monocular RGB image through a mixed attention depth coding module, and obtaining the hierarchical depth feature representation; carrying out autoregression depth decoding through a self-adaptive edge perception depth decoding module to generate an initial depth map; a depth confidence map is calculated through a geometric consistency constraint optimization module and is fed back to a coding module for iterative optimization, and a refined depth map is output; and three-dimensional Gaussian ellipsoid scene representation is constructed through the Gaussian ellipsoid scene reconstruction module. According to the invention, high-precision depth estimation and high-quality three-dimensional reconstruction are realized by constructing a depth-coupled closed-loop cooperative system.

The invention discloses a single view reconstruction and rendering method, which belongs to the field of view reconstruction and rendering, and comprises the following steps of: firstly, generating multi-view feature representation with strong geometric consistency from a single input image by introducing an image diffusion module of a cross attention mechanism; a point cloud reconstruction module with self-attention and cross-attention is utilized, and multi-view information is fused to reconstruct an accurate three-dimensional point cloud; and finally, constructing differentiable three-dimensional Gaussian representation based on the point cloud, rendering the differentiable three-dimensional Gaussian representation, and outputting a new view angle image, a normal map and a depth map. According to the system, a staged training strategy is adopted, and a composite loss function including multi-scale bidirectional consistency smooth loss and feature consistency loss is innovatively used for optimization. According to the method, the problems of low geometric accuracy, multi-view inconsistency, detail missing and the like in single-view reconstruction are effectively solved, and the method can be widely applied to the fields of virtual reality, digital twinning, cultural heritage digitization and the like.

The invention belongs to the technical field of intelligent measurement, and discloses a mobile terminal automatic measurement method and system based on YOLO key point detection and an AR platform, and the method comprises the steps: obtaining multi-modal data, and carrying out the preprocessing; performing target detection and key point positioning based on the RGB image to obtain two-dimensional key point coordinates; based on the depth image and the pose information, a three-dimensional space coordinate system is established, the two-dimensional key point coordinates are converted into the three-dimensional space coordinate system, and in the conversion process, a self-adaptive weight fusion mechanism is adopted to perform fusion on the multi-modal data to obtain a fusion result; a nonlinear error propagation control model is adopted to suppress error accumulation in the process of converting the two-dimensional coordinates into the three-dimensional coordinates; and calculating morphological parameters of the target object based on the converted three-dimensional key point coordinates. According to the invention, the problems of low measurement efficiency, poor precision, high equipment cost, insufficient cross-platform adaptation and the like in the prior art are solved, and high-precision, real-time and low-cost automatic measurement of various types of target objects at the mobile terminal is realized.

The invention belongs to the technical field of robot positioning and three-dimensional mapping, and discloses a high-precision map reconstruction method and system based on monocular vision, a medium and equipment. Scene image data are acquired through a monocular image acquisition module, after feature extraction and matching are performed on each frame of image, attitude information acquired by an inertial measurement unit is fused, and pose calculation of a robot is completed through a sparse vision SLAM system; meanwhile, an image dense depth map is generated by a monocular depth estimation model based on an attention mechanism, and the image dense depth map is converted into a single-frame color dense point cloud in combination with image RGB color information. According to the invention, a loose coupling fusion strategy is adopted to carry out spatial registration on a single-frame colored dense point cloud and a robot pose at a corresponding moment, multi-frame data fusion is completed through point cloud splicing and optimization, and a globally consistent three-dimensional dense point cloud map is constructed to realize scene modeling. The method has the characteristics of high robustness and high reconstruction precision, and can be effectively applied to three-dimensional map construction in an outdoor complex environment.

The invention belongs to the technical field of steel reinforcement framework three-dimensional reconstruction, and particularly discloses an automatic high-precision three-dimensional dense reconstruction method for a box girder steel reinforcement framework, which comprises the following steps of: dividing a scanning area into a plurality of sub-areas according to a depth image by moving an inspection robot along the steel reinforcement framework, and performing multi-angle local scanning in each sub-area; constructing a local point cloud model by combining camera poses during acquisition, further calculating a relative spatial transformation relationship by using an overlapping region between adjacent local models, constructing a pose map and implementing global optimization, and uniformly adjusting the poses of the local models in a world coordinate system; and finally, all corrected local point clouds are fused to generate a complete global three-dimensional model, so that collaborative reconstruction operation of domain-divided scanning, segmentation reconstruction and global optimization is realized, frame-by-frame accumulation of errors in traditional scanning-while-splicing reconstruction is effectively blocked, long-distance drift is remarkably inhibited, and the precision of three-dimensional reconstruction of the reinforcement cage is greatly improved.

The invention relates to the technical field of visual navigation, in particular to a blind person navigation path planning method combining visual SLAM and semantic segmentation. The method comprises the following steps: acquiring an environment image and pose data; performing front-end tracking by using the pose data, and determining a key frame sequence; generating a depth map based on the environment image, and identifying a passable area according to the depth map; constructing a three-dimensional point cloud map according to the key frame sequence; semantic segmentation is carried out according to the three-dimensional point cloud map, and obstacle type labels are recorded; according to the obstacle type label, carrying out safety level layering on the passable area, and determining a path cost weight; performing global path planning based on the path cost weight, and generating a semantic enhancement navigation trajectory; and driving real-time voice guidance by using the semantic enhancement navigation trajectory, and calculating a path execution deviation corresponding to the real-time voice guidance. According to the method, obstacle types are distinguished through semantic segmentation based on a visual navigation technology, a semantic enhancement path is generated, semantic hierarchical navigation is realized, and the intelligence of path decision is improved.

The invention discloses a video stream defogging method and system for 5G remote control, and relates to the technical field of image enhancement. The method comprises the following steps: acquiring a foggy video in real time; inputting the single-frame foggy video into a pre-trained monocular depth estimation neural network, and reasoning to obtain a depth map with the same size as the input single-frame foggy video; performing close-shot and long-shot region division on the scene of the current foggy video frame according to the depth map to obtain a region mask with the same size as the depth map; based on the region mask and the foggy video, calculating to obtain an atmospheric light parameter; calculating the transmissivity of each pixel according to the distance estimation result of each pixel in the depth map, and obtaining a transmissivity map with the same size as the depth map; based on the depth map, the atmospheric light parameters and the transmissivity map, adaptive calculation is performed on the foggy video to realize defogging, and a defogged clear video frame is obtained; the method can adapt to different depth-of-field fog effects, realizes different depth-of-field defogging, and outputs clear video frames.

The invention discloses an agricultural product maturity detection method and system, an electronic device and a storage medium, and relates to the technical field of agricultural product maturity detection.The method comprises the steps that a TMMFNet model is constructed, and in the TMMFNet model, a feature extraction module extracts modal features of an RGB image, a near-infrared NIR image and a Depth image of the same agricultural product object; and the cross-modal feature processing module processes each modal feature and outputs an RGB enhanced feature, an NIR enhanced feature and a Depth enhanced feature. And the modal adaptive fusion module adaptively calculates a weight according to the global statistical information of each enhanced feature and performs weighted fusion to obtain a fused feature, and a classifier processes the feature to output a maturity category. And then training a TMMFNet model, and performing maturity discrimination on the input multi-modal image by using the agricultural product maturity detection model obtained by training. According to the invention, the maturity discrimination precision can be effectively improved.

The invention provides a personal question and answer method based on an observation-recording-decision mechanism, which comprises the following steps: an intelligent agent captures RGB images and depth images from all directions through multi-view perception, is used for constructing a 3D scene graph and mapping the 3D scene graph to a 2D semantic map, and meanwhile, the intelligent agent marks each passing position on the 2D semantic map, so that the 3D scene graph is mapped to the 2D semantic map; and dynamically updating the weight of a non-visited position, reducing the selection probability of a boundary point in a marked passing region, based on a 2D semantic map in an observation stage, distinguishing whether a question can be answered or not by an intelligent agent according to an observed RGB picture, if so, directly generating a response, otherwise, navigating to a new region, and repeating the steps until an available or maximum step number is reached, and completing the question answering. According to the method, through construction of a semantic map, weight regulation and control navigation, and fusion design of special VLM analysis and double-criterion decision making, decision making of agent non-redundancy exploration and accurate question and answer is achieved.

The invention relates to the field of industrial defect detection, in particular to an industrial defect detection self-supervised segmentation method for iterative pseudo label refinement, which comprises the following steps of: constructing a system comprising a strategy model, a refinement model, a reward model and a meta-learning training module; inputting the multi-modal data into the strategy model, and outputting a rough defect mask; based on the rough defect mask, prompting refinement is carried out through a refinement model, and a refinement mask is output; based on the refinement mask, the to-be-detected product image, the standard template image, the depth image and the infrared image, calculating a comprehensive quality score through a reward model, and screening high-quality samples with qualified scores; and updating a preset training data set based on the high-quality sample, performing supervised training on the strategy model by using the updated training data set, and repeating the steps to form an iterative loop. By constructing a self-supervised closed loop, a system can be driven to autonomously learn defect features from an unlabeled production line multi-modal image only by a small amount of initial reference data.

The invention provides an AI human shape recognition perception system and method based on binocular vision, and the method comprises the steps: carrying out the real-time collection through a binocular camera when a doorbell key is triggered, and carrying out the preprocessing of an original image collected in real time; performing coarse parallax estimation on the real-time image rectification to obtain a full-field coarse depth map, determining a human shape candidate region list by using the full-field coarse depth map and combining the heat source region of interest, and performing fine parallax estimation on the human shape candidate region list to obtain a fine depth patch; converting the corresponding fine depth patch into a three-dimensional point cloud set according to the pose information, and performing scale prior screening based on the corresponding three-dimensional point cloud set to obtain a plurality of human shape candidate reserved areas; and performing fusion identification according to the extracted multi-modal features to obtain a human shape identification result. According to the technical scheme provided by the invention, layered parallax estimation and three-dimensional scale prior screening can be carried out on the real-time image to realize high-reliability human shape recognition under the condition of low power consumption, so that the recognition reliability of a sensing system is improved.

The invention provides a camera pose estimation method based on 2D Gaussian splash. The camera pose estimation method comprises the following steps: acquiring a depth map and a normal map from a training image pose based on a pre-trained 2D Gaussian splash model; acquiring a grid ray starting point based on the depth map; generating a main ray and a hemispherical ray for each sampling point based on the normal diagram and the ray starting point; constructing a ray and image matching network and a lightweight convolutional neural network, and performing network training through a loss function; estimating the position of the camera through the trained ray and image matching network and the lightweight convolutional neural network based on the ray features of the main ray and the hemispherical ray and the image features of the query image, and constructing a rotation matrix to realize initial camera pose estimation; and optimizing the initial camera pose by adopting a depth-guided pose optimization method to obtain a final optimized camera 6D pose estimation result. The method gets rid of the dependence of an initial value, and has the advantages of high precision, strong robustness, efficient calculation and wide application.

The invention discloses a reinforcing steel bar surface binding point identification and positioning method based on point cloud and pixel mapping. The method comprises the following steps: obtaining original three-dimensional point cloud data to carry out outlier noise point elimination; performing plane fitting on the denoised point clouds, identifying template surface point clouds and removing the template surface point clouds; mapping the spatial physical coordinate attribute of the steel bar surface point cloud into a three-channel coding image; separating the three-channel coded image into three single-channel images: a length image, a width image and a depth image; processing the depth image to extract an initial reinforcing steel bar straight line set; screening out a parallel steel bar straight line set from the initial steel bar straight line set according to a set reference direction and a parallel angle tolerance threshold value; and calculating straight line intersection points in the parallel steel bar straight line set to obtain candidate binding point pixel coordinates, determining representative binding point pixel positions through clustering, querying coordinate information of the three single-channel images, and outputting three-dimensional space coordinates of the steel bar binding points. According to the invention, robust and accurate steel bar binding point identification and positioning are realized.

The invention provides a box girder defect recognition method and system based on structured light and deep reinforcement learning, and relates to the technical field of box girder defect recognizing.The method comprises the steps that equipment integration and joint calibration are conducted on a to-be-detected box girder component, structured light projection is conducted according to a preset track, and a to-be-detected box girder component is obtained; the method comprises the following steps: triggering an event camera to image by utilizing controlled stroboscopic scanning of structured light, obtaining an event stream and a structured light depth map, carrying out representation conversion and multi-modal alignment, obtaining multi-modal mapping alignment data to carry out primary defect identification, obtaining an initial defect identification result, and triggering a deep reinforcement learning active perceptron to carry out perception strategy analysis. And obtaining a perception strategy analysis result, and carrying out deepening defect identification on the to-be-detected box girder component to obtain a defect identification result. The technical problems that in the prior art, box girder defect detection is low in detection efficiency, not high in accuracy and poor in detection comprehensiveness are solved. The technical effect of improving the efficiency, accuracy and comprehensiveness of box girder defect detection is achieved.

The invention discloses a bridge disease spatial form quantitative characterization method based on fusion of a three-dimensional laser point cloud and a two-dimensional image. The method comprises the following steps: synchronously acquiring image data and laser point cloud data of a bridge disease; the data sum is preprocessed; constructing a PSAG-Net model, and inputting the point cloud data into the PSAG-Net model for training; constructing a CM-FPN model, and inputting the image data into the CM-FPN model for training; segmenting the crack point cloud data by using a PSAG-Net model, carrying out post-processing on the completely segmented large-scale crack point cloud, complementing the missing crack region, calculating the depth value of the large crack, and counting the depth distribution; and for the small-scale crack point cloud which is not completely segmented, converting the small-scale crack point cloud into a depth map and carrying out binarization, carrying out crack pixel-level detection and segmentation by using a CM-FPN model, carrying out back projection to a three-dimensional coordinate system, and calculating a depth value. According to the invention, accurate quantitative characterization of depth information is realized.

The invention discloses a three-dimensional semantic scene completion method based on camera enhancement, a medium and equipment, and the method comprises the steps: obtaining a depth map and an optical flow graph through a left image and a right image, extracting a two-dimensional feature from the left image, and extracting a two-dimensional feature from the depth map; mapping the left image to obtain context features; performing depth optimization based on the two-dimensional features, the two-dimensional features and the depth map to obtain depth estimation distribution; obtaining and generating an expanded three-dimensional feature map through context feature operation expansion, and sequentially executing three-dimensional deformable cross attention and deformable self-attention operations on the three-dimensional feature map to output updated three-dimensional features; and geometric semantic enhancement is carried out on the updated three-dimensional features, and finally three-dimensional semantic scene completion is completed. The depth prediction precision is improved, depth estimation optimization is carried out, finally the geometric structure and detail prediction capability of the model is enhanced, and three-dimensional semantic scene completion is completed.

Disclosed herein is a method for estimating a depth of an image. The method may include acquiring a first depth image based on left and right images collected from a stereo camera, estimating a second depth image based on any one of the left and right images, generating a depth estimation model based on first depth values within a depth range recognizable by the stereo camera in the first depth image and on second depth values of the second depth image matching the first depth values, and inputting third depth values of a range intended to be estimated in the second depth image to the depth estimation model, thereby estimating fourth depth values of the first depth image matching the third depth values.

The invention discloses an unmanned aerial vehicle motion planning method and system for guiding visual heat conduction based on depth information, and the method comprises the steps: constructing a motion planning model, and inputting a depth image, an unmanned aerial vehicle attitude and an expected speed into the motion planning model, and real-time high-speed obstacle avoidance of the quad-rotor unmanned aerial vehicle in an unknown complex environment is realized. According to the motion planning model, a visual heat conduction module with a global receptive field is adopted as a sensing system trunk, a depth information guide heat conduction operator module is introduced, depth information is coded into an energy weight, and heat conduction calculation is performed after spatial information and scene representation are guided to be coupled, so that the model can focus on a close-range obstacle. The generated depth information guide scene representation is then input to a decision module to generate an action instruction.

The invention provides an intelligent vision-guided mechanical arm precise grabbing system. The system comprises a bionic compound eye vision module for collecting multi-angle image information of a target object to generate a dynamic depth map, and a bionic touch sensing module for collecting touch information at the tail end of a mechanical arm to generate a pressure distribution heat map. And the dynamic self-optimization grabbing strategy module is used for generating a grabbing strategy by utilizing the two graphs. And the real-time trajectory planning and obstacle avoidance module optimizes a grabbing path according to the grabbing strategy, the environment obstacle information and the mechanical arm state data, and generates an optimal strategy. The robot can sense the movement of a target object and the change of environmental obstacles in real time, dynamically adjust the force and the posture in the grabbing process, and optimize the obstacle avoidance path in real time. Compared with the prior art, the system has the advantages that the aspects of dynamic obstacle avoidance, target object motion compensation, multi-modal information fusion robustness and the like are remarkably improved, the problem of insufficient adaptability in complex dynamic scenes in the prior art is solved, and high success rate and stability of a precise grabbing task under complex conditions are ensured.

The invention relates to the technical field of autonomous navigation of unmanned cleaning trolleys, in particular to a robot autonomous navigation coordination control method and system based on an ST-TD3 strategy. The training action decision model generates actions based on the current state of the unmanned cleaning trolley, and the unmanned cleaning trolley executes the actions to adjust the speed and the yaw angle; the state comprises an environment depth characteristic and an obstacle density characteristic of an environment where the unmanned cleaning trolley is located, and a relative distance and a relative azimuth angle between the unmanned cleaning trolley and the target point; environment depth features and obstacle density features are extracted based on depth image features acquired by an unmanned cleaning trolley; the actions comprise linear velocity and angular velocity, the linear velocity and the angular velocity respectively meet the linear velocity maximum value constraint and the angular velocity maximum value constraint of the unmanned cleaning trolley, and the angular velocity represents the deflection direction through positive and negative values. The unmanned cleaning trolley can efficiently, safely and stably run in a complex cabin environment.