4945 results about "Rgb image" patented technology

A system and method are disclosed for generating hyperspectral images from multi-modal sensor data including RGB, LiDAR, thermal, and near-infrared inputs. Training data includes hyperspectral images and corresponding multi-modal measurements. Spectral band grouping is performed based on correlation coefficients. A multi-modal decomposition network with cross-modal attention mechanisms generate reconstructed hyperspectral images by fusing complementary sensor information. A fine-tuning network creates reconstructed RGB images. A comprehensive quality assurance system analyzes spectral consistency, cross-modal coherence, and fusion artifacts to generate quality metrics. Missing data compensation strategies handle corrupted sensor inputs using information from other modalities. The system includes temporal integration for video sequences and multi-resolution processing for different sensor resolutions. Quality metrics guide network weight adjustments to improve reconstruction accuracy while maintaining robustness to sensor failures and environmental variations.

The present invention discloses an edge-guided RGBD underwater salient object detection method with multi-attention, which includes the steps of: S1 performing channel-dimension preprocessing on RGB image and depth map and serially connecting them to obtain an RMD image; S2 inputting the RMD image into a trained salient object detection model including an encoder structure and a decoder structure; and S3 obtaining salient images and edge images output from the salient object detection model as object detection results, in which the encoder generates feature maps with reduced resolution layer by layer using the RMD image based on a residual neural network, and the decoder structure includes an edge detection branch used to fuse the feature maps layer by layer, and a saliency detection branch used to fuse the feature maps and edge feature maps layer by layer. The present invention preprocesses the input image with underwater imaging features, and uses an edge prediction image to guide multi-layer salient object detection, thereby enhancing edge details of the salient image.

The invention discloses an exhibition hall robot visual language navigation method based on a large model, belongs to the field of exhibition hall robots, and aims at solving the problems existing in an existing exhibition hall robot. According to the method, an exhibition hall robot multi-mode scene sensing system senses a current scene, and candidate target objects are obtained through fusion features of RGB images and laser point clouds; extracting image features of all candidate target objects in the scene; according to the exhibition hall robot multi-mode environment understanding system, an exhibition hall robot receives a voice instruction, a visual language multi-mode large model is used for understanding a target object and a user intention, then high-dimensional semantic features of a target object text are extracted from the system, and the target object is found out from candidate target objects; according to the exhibition hall robot intelligent navigation system based on the multi-modal large model, an exhibition hall robot carries out path planning and autonomous movement according to a pose sequence generated by an intention, and self behaviors are dynamically adjusted according to a real-time changing scene provided by the exhibition hall robot multi-modal scene sensing system.

A multi-scale cascaded hourglass depth map completion method guided by RGB image is presented. [Solution] The multi-scale cascade hourglass depth map completion method includes step S1 of obtaining a sparse depth map to be completed and a corresponding RGB color guide image, performing channel-dimensional pre-processing on the sparse depth map and the RGB color guide image and connecting them in series to obtain a target RGB_D image to be processed, step S2 of inputting the target RGB_D image to be processed into a trained depth map completion network model including an early fusion encoder, a sparse depth map teaching module, a multi-scale hourglass completion module and an optimization reinforcement module, and step S3 of obtaining the output of the optimization reinforcement module as a completed dense depth map.

The invention relates to an automatic driving vehicle path planning method and device based on a large language model, equipment and a medium. The method comprises the steps that a traffic scene RGB image, a traffic scene depth image, laser radar point cloud data and inertial measurement unit data of an automatic driving vehicle are acquired; performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the laser radar point cloud data and the inertial measurement unit data to construct multi-modal traffic scene information; calling a preset large language model to determine the motion state of the autonomous vehicle and natural language description of traffic targets around the autonomous vehicle according to the multi-mode traffic scene information; and determining the attention weight of the target node by adopting an attention mechanism in a large language model according to the natural language description of the traffic target, and determining the optimal driving path by adopting a preset path planning algorithm according to the attention weight of the target node, the motion state of the autonomous vehicle and the target position. According to the invention, the adaptive capability of path planning is obviously improved.

The invention discloses a robot grabbing posture generation method and a related device, and the method comprises the steps: obtaining an RGB image, a depth image and a position coordinate of a target object, and generating a spatial calibration matrix through data preprocessing; target object segmentation is carried out on the calibrated RGB matrix according to the target point coordinate sequence, a segmentation mask is generated, and object mass center coordinates are calculated; generating a three-dimensional point cloud by using the calibrated depth matrix, the segmentation mask and the camera parameters, extracting a plane point set and a non-plane point set through an RANSAC algorithm, and analyzing to obtain an axial feature vector and a plane normal vector; finally, grabbing parameters are calculated according to the feature vectors and the centroid coordinates, and a grabbing posture transformation matrix is generated through vector operation. According to the technical scheme, under the condition of not depending on a preset model library, the grabbing posture of an unknown object is generated by analyzing the geometrical characteristics of the object, the accuracy problem in the process of converting two-dimensional image information into three-dimensional grabbing parameters is solved, and the adaptability and reliability of a robot grabbing task are improved.

The invention relates to the technical field of industrial robots, in particular to an industrial robot disordered grabbing system and method based on multi-modal sensing, and the system comprises a multi-modal sensing module, a pose estimation and correction module, a grabbing task planning module and a mechanical arm execution module. The multi-modal sensing module is used for acquiring RGB image data, depth image data, point cloud data and force / torque data in a working environment, and the pose estimation and correction module performs feature extraction and fusion on multi-modal data based on topology invariant mapping and performs adaptive pose estimation on a target object. The grabbing task planning module generates an optimal grabbing strategy based on deep reinforcement learning and outputs a grabbing control instruction, and the mechanical arm execution module receives the grabbing control instruction, controls a mechanical arm to execute a grabbing action and monitors a grabbing state through force / torque feedback. And the environmental adaptability and the perception robustness are improved.

An aero-engine augmented reality virtual-real fusion method based on depth prior scene reconstruction comprises the following steps: firstly, collecting data of an aero-engine look-around RGB image and a depth image, calculating a camera frame pose by using an SFM algorithm and obtaining a sparse point cloud, and optimizing the camera pose by using an ICP algorithm and the point cloud converted from the depth image to obtain an aligned prior dense point cloud; gaussian points are initialized based on the priori dense point cloud and the sparse point cloud, and random generation of 3D Gaussian is limited by using the boundary of the dense point cloud as geometric constraint; through the three-dimensional reconstruction of the adjacent multi-view enhancement strategy, the spatial and visual consistency is improved, and an aero-engine model is reconstructed; then understanding the reconstructed aero-engine model based on semantic segmentation, identifying and marking core components, and finally performing virtual-real fusion visualization on the reconstructed aero-engine through augmented reality to guide training, assembling and maintenance operations. According to the invention, rapid high-quality three-dimensional reconstruction and augmented reality virtual-real fusion visualization of the aero-engine are realized.

The invention relates to a human shape posture recognition method and system based on image analysis, and the method comprises the steps: inputting the reference human shape data of a monitored object, and collecting the multi-modal data of an RGB image, an infrared image and inertial measurement data in a target scene in real time; space-time alignment processing is carried out, and corresponding features are fused; image space features are extracted, time sequence modeling is carried out on inertial data, and dynamic weighted fusion of two paths of network outputs is realized through a gating mechanism; human body basic joint points are positioned, and refined posture vectors including joint angles and limb relative positions are generated; according to attitude data and environment information collected in real time, adaptively adjusting an attitude classification threshold value and a similarity measurement standard, and predicting abnormal behaviors in a future time period; and when the abnormal behavior is predicted, triggering to execute a preset safety measure. Multi-modal data can be efficiently fused, attitude features can be accurately extracted, an identification strategy can be adaptively adjusted, and human shape attitude identification with behavior trend prediction capability can be realized.

Disclosed in the present invention are a hyperspectral calculation imaging method and system based on multi-domain fusion of spatial, spectral and frequency domains, and a medium. The method of the present invention comprises: using a two-dimensional offline discrete cosine transform (DCT) to convert an RGB image Y into a frequency domain to obtain a frequency domain feature map Yfreq; extracting a frequency information image (I) from the frequency domain feature map Yfreq; using a two-dimensional offline inverse discrete cosine transform (IDCT) to transform the frequency information image (I) into a spatial domain to obtain a frequency information image Xfreq of the spatial domain; and fusing the frequency information image Xfreq of the spatial domain into the spatial-spectral domain features of the RGB image Y to generate a hyperspectral image (II). The present invention aims to solve the problems of poor detail information and low reconstruction accuracy of hyperspectral images in existing hyperspectral calculation imaging, and realizes high-fidelity reconstruction of a target spectrum.

The invention provides an unmanned aerial vehicle identification early warning method and system. The method comprises the following steps: acquiring RGB image data, thermal radiation data and spectral data of an unmanned aerial vehicle no-fly zone through a visible light camera, an infrared thermal imager and a multispectral imager; performing transmission preprocessing on the RGB image data, the thermal radiation data and the spectral data, and adaptively adjusting multi-level fusion of a fusion weight based on real-time environmental parameters to generate target fusion data; based on a deep learning model and a tracking prediction algorithm, performing unmanned aerial vehicle identification early warning on the target fusion data, and generating early warning data; and transmitting the target fusion data and the corresponding abnormal event log to a cloud server, and updating the deep learning model by adopting the target fusion data and the abnormal event log. Through cooperative work of a multi-mode sensor, visible light, infrared, multispectral and other wave bands are covered, all-weather and full-scene unmanned aerial vehicle detection is achieved, the fusion weight is adjusted in real time based on real-time environment parameters, and the accuracy of the recognition result under the complex air situation is ensured.

The invention discloses a multi-modal visual fusion complex scene small target detection tracking method and system, and relates to the technical field of unmanned aerial vehicle target tracking, and the method comprises the steps: employing a visible light camera, an infrared thermal imager and a laser radar sensor which are carried on an unmanned aerial vehicle platform, and synchronously collecting RGB images, thermal infrared images and point cloud data; the consistency of the multi-modal data is ensured through data preprocessing and space-time alignment; constructing a lightweight double-branch network to extract multi-scale features, generating a fusion feature map by adopting adaptive weighted fusion, and generating depth information by utilizing point cloud to assist in scale estimation; a small target detection head is designed based on the fusion feature map, and precise detection is realized in combination with a feature pyramid network, adaptive scale prediction and a context awareness suppression mechanism; furthermore, through multi-mode cooperative tracking, including target association, spatio-temporal context modeling, trajectory prediction and a re-detection mechanism, tracking continuity is ensured.

The invention discloses an event and RGB image fused SNN target tracking method and system, and the method comprises the steps: converting an event flow from an event camera into an event frame image through a representation method based on time stacking, and enabling the event frame image to be aligned with an RGB frame image from an RGB camera in time and space; performing target detection on the event frame image # imgabs0 # and the RGB frame image # imgabs1 # after alignment by using a spiking neural network SNN, wherein the spiking neural network SNN comprises a feature extraction module, a feature fusion module and a detection head; and learning the similarity of the targets by using the anchor frame extracted by the spiking neural network SNN and the fusion feature map # imgabs2 # to perform multi-target tracking. The invention aims to realize fusion of the fusion event and the RGB image so as to ensure high precision and high efficiency of multi-target tracking, and keep stable detection performance and tracking performance in static and high-motion scenes.

The embodiment of the invention provides an unhooking robot control method based on multi-source visual fusion, which is applied to the technical field of robot control and comprises the following steps: acquiring an RGB image, a depth image, an infrared image and IMU data through a multi-source sensing system mounted at the tail end of a robot; carrying out feature fusion identification by adopting a double-branch neural network, and outputting the boundary contour of the lifting hook and the three-dimensional coordinates of the optimal grabbing point; the visual coordinates are unified to a robot base coordinate system through a registration correction mechanism; a Transform prediction model is constructed based on the visual and inertial signals, and future pose changes of the lifting hook are estimated; a feedforward control track is generated to counteract swing of the lifting hook, and track correction is carried out in combination with visual servo feedback; and a joint instruction is generated through path planning and inverse kinematics solution, and the mechanical arm is driven to complete precise unhooking operation. According to the method, the recognition precision, the anti-interference capability and the operation success rate of unhooking operation in complex illumination and dynamic environments are effectively improved.

The invention relates to the technical field of crop disease recognition, in particular to a crop early disease recognition method and system based on time sequence feature fusion, and the method comprises the steps: obtaining an RGB image and a temperature feature image of a target leaf, and collecting multispectral feature data through a spectral imaging device; based on the time sequence image sequence, generating a coordinate transformation matrix of the first frame image by using a PnP algorithm, and aligning subsequent images according to the coordinate transformation matrix; multi-modal features are extracted by using a deep neural network, and the deep neural network comprises a 3D convolutional layer and an adaptive attention mechanism and is used for extracting features of space and time dimensions; realizing cross-modal feature fusion through a graph convolutional network based on the spectral data and the multi-modal features; inputting the fused features into a classifier, and generating the disease category and the disease severity of the target leaf based on the output of the classifier; outputting the disease transmission risk and the early warning result, and capturing the tiny dynamic change in the disease development process.

The invention discloses a soil fertility intelligent assessment method and system based on image recognition, and relates to the technical field of data processing, and the method comprises the steps: obtaining land image information of a to-be-assessed region, the land image information comprising a multispectral image, a thermal imaging image and an RGB image of the to-be-assessed region; carrying out image preprocessing on the land image information, and carrying out vegetation coverage and health assessment according to the multispectral image: speculating soil characteristics according to the RGB image: speculating the humidity change of a soil surface layer according to the temperature distribution of a thermal imaging image, and identifying a drought region and a region with sufficient moisture; integrating the high-resolution texture features of the RGB image, the spectral information of the multispectral image and the speculation result of the temperature distribution of the thermal imaging image to generate a comprehensive analysis result; and classifying soil types and evaluating fertility grades according to a comprehensive analysis result. The problem that in the prior art, in intelligent soil fertility evaluation, workers need to carry out on-site sampling examination is solved, and the technical effect of improving the soil fertility evaluation efficiency is achieved.

The invention discloses a mechanical arm motion control method based on multi-agent cooperation, and the method comprises the steps: firstly, receiving an RGB image through a sub-task generation agent, and generating a structured sub-task sequence according to a natural language task instruction of the RGB image; secondly, performing joint modeling on a task text and a scene image through a 3D sensing intelligent body, positioning specific coordinates of a target object in a three-dimensional space, reasoning dynamic characteristics of a current environment based on historical state information of a robot by combining an environment sensor, and generating an environment sensing vector; and finally, the action generation agent performs fusion modeling according to the subtask text, the subtask target coordinates, the current state of the robot and the environment perception vector, generates a continuous action vector, drives a mechanical arm to complete each subtask action, and constructs closed-loop feedback by a controller and a discriminator to realize task execution state judgment and automatic circulation. The precise action control instruction can be effectively generated, and the task execution fineness of the mechanical arm is remarkably improved.

The invention provides a semantic vision SLAM method based on depth mask segmentation in a dynamic environment. The semantic vision SLAM method comprises the following steps: inputting an RGB image and a depth image into a tracking thread for preprocessing; inputting the preprocessed RGB image into a semantic segmentation thread for precise detection and segmentation of a scene target, obtaining a semantic tag of each pixel, and generating a segmentation mask; in combination with the depth information consistency of the preprocessed depth image, correcting and optimizing the mask to obtain a mask image; inputting each frame of corrected mask image into a dynamic feature screening thread, and determining dynamic feature points of potential dynamic objects; epipolar constraints are constructed through matching points in a background region of the RGB image, and dynamic feature points in a high dynamic object and a potential dynamic object in the RGB image are eliminated; and merging and splicing local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated through a point cloud mapping thread, and constructing a global three-dimensional point cloud map without the dynamic objects. According to the method, the mapping precision is improved.

The invention relates to the technical field of remote sensing monitoring, in particular to a remote sensing target detection method and system for a low-visibility image. The method comprises the following steps: acquiring multi-modal remote sensing image data; carrying out defogging enhancement processing on the low-visibility input image; normalizing the defogged RGB image and the defogged IR image, and then splicing and fusing the RGB image and the IR image; carrying out layer-by-layer coding on the multi-modal fusion image by adopting a mixed trunk structure fusing Transform, Mamba and CNN (Convolutional Neural Network); performing frequency domain decomposition on the trunk output features based on two-dimensional wavelet transform; generating an HR feature map by adaptively selecting a key region; and carrying out cross-scale aggregation on the HR feature map to obtain a detection target frame. Through the multi-modal image defogging enhancement and feature distillation mechanism, the definition and contrast of the remote sensing image in severe weather such as haze and rainy days are effectively enhanced, the shielding interference of environmental degradation on small target detection is weakened, and the stability and adaptability of the model in complex weather scenes are enhanced.

The invention belongs to the technical field of tree three-dimensional reconstruction, and particularly discloses a tree neural rendering and reconstruction method based on a sparse view, and the method comprises the steps: collecting an RGB image of a tree through a camera, obtaining camera parameters, inputting the camera parameters into a pre-trained depth estimation model, generating a rough depth map, and inputting the rough depth map into a neural radiation field network; introducing a cone sampling strategy, integrating a position coding method, a coding annealing strategy, depth sorting and depth consistency constraint, and extracting point cloud and picture features in the picture; carrying out implicit surface reconstruction based on volume rendering; introducing color constraint and geometric constraint based on color deviation and geometric deviation of implicit surface reconstruction; and a three-dimensional model of the tree is reconstructed from the implicit surface by using a Marking Cubes algorithm. By adopting the technical scheme, tree rendering is carried out based on the neural radiation field, tree reconstruction is carried out based on body rendering, and the reconstructed tree model can better retain and reproduce original geometric features and real textures of the tree.

The invention discloses a weld joint key point recognition method, device and equipment, a medium and a program product, and the method comprises the steps: generating a data set based on training image data and annotation data, constructing an original recognition model, improving the original recognition model, and training and optimizing the improved original recognition model based on the data set. Obtaining a target recognition model; acquiring a depth image and an RGB image of the welding part through image acquisition equipment on the welding robot; the RGB image is input into the target recognition model for weld joint key point recognition, weld joint key point coordinates of the welding part are determined based on the recognition result and the depth image, accurate recognition of the weld joint key points is achieved, a welding robot is guided to accurately reach the weld joint key point coordinate positions for welding, key point recognition autonomy and adaptability are improved, and the welding quality is improved. And welding tasks with different requirements are effectively adapted, the welding efficiency is greatly improved while the welding quality is ensured, and the time cost is reduced.

The invention relates to the technical field of three-dimensional model reconstruction, and discloses a three-dimensional model reconstruction method based on point cloud data processing, which comprises the following steps: step 1, point cloud data acquisition and multi-modal fusion: acquiring point cloud data through a laser radar, an RGB camera and multi-view three-dimensional reconstruction to obtain a point cloud set, aligning the point cloud data with an RGB image to obtain a point cloud image set; establishing a mapping relation between point cloud coordinates and image pixel coordinates through an internal parameter matrix and an external parameter matrix of the camera to obtain aligned multi-modal point cloud data; 2, preprocessing the point cloud data: performing cleaning and downsampling processing on the multi-modal point cloud data obtained in the step 1; by combining point cloud data obtained by a laser radar, an RGB camera and multi-view three-dimensional reconstruction, accurate alignment of different sensor data is realized, high-precision and complete multi-modal point cloud data is obtained, blind areas of all sensors are effectively supplemented, and the precision and reliability of a three-dimensional reconstruction result are improved.

Provided in the present invention are an interaction and generation method for obtaining a three-dimensional model by means of photographing, a method for obtaining a picture by means of a three-dimensional model, and a part machining and manufacturing method. The interaction and generation method comprises: a user using a terminal to capture scene information, wherein the scene information can be an outdoor or indoor scene, the terminal can be a smart phone, an AR / VR device or other smart terminals having a photographic function, and the captured scene information is RGB and / or RGBD (including depth) information; and sending an RGB image and / or RGBD image I to a monocular three-dimensional reconstruction algorithm, so as to obtain a corresponding three-dimensional model, which can be displayed on an interaction interface and superimposed with a real scene, or can be displayed on a single page.

The invention discloses a grabbing posture generation method and system based on a multi-modal large model, and the method comprises the steps: carrying out the cross-modal matching of visual features and semantic features in the multi-modal large model when a voice instruction and an RGB image are inputted, and obtaining the position information of a control function code and a target object; when an RGB image with a hand drawing instruction is input, obtaining position information of a control function code, a target object and a path point; calculating the point cloud data of the target object according to the position information of the target object and the depth information, inputting the ideal point cloud of the target object into a target recognition network model after preprocessing, carrying out the grabbing region recognition of the point cloud of the target object region, outputting a region with high grabbing confidence, and mapping a real coordinate system; constructing a point cloud bounding box, and generating a grabbing posture candidate set; the grabbing posture with the highest quality is selected as the grabbing posture of the robot by calculating the grabbing posture candidate score; and executing a target grabbing task in combination with the control function code and the grabbing path.

A method and apparatus for reconstructing a three-dimensional (3D) scene from a two-dimensional (2D) input image of the scene using a fully-differentiable transformer-based encoder-decode. A 2D input image encoded into a set of image features using a pre-trained vision transformer model, wherein the vision transformer model is pre-trained with multi-view RGB image supervision and point cloud supervision. The set of image features is projected onto a 3D triplane representation using a transformer decoder to obtain output triplane tokens. A triplane representation is created from the tokens and queried. 3D point features of color and density for volumetric rendering re predicted using a multi-layer perceptron. The geometry of the generated 3D asset is represented with a surface mesh including vertices and triangular faces. A texture map by is created with a multichannel image in UV space. Multiple views of the 3D scene are simultaneously generated based on the surface mesh.

The invention discloses a three-dimensional scene reconstruction method and device based on structure perception. The method comprises the following steps: constructing a sparse point cloud generation model according to a diffusion model based on a Unet architecture; inputting the information into a sparse point cloud generation model, and generating camera internal and external parameters and sparse point cloud data of the RGB image frame under the corresponding view angle; constructing a rough three-dimensional structure grid according to the sparse point cloud data; carrying out random sampling on the rough three-dimensional structure grid, and carrying out Gaussian field initialization on the Gaussian primitives according to a random sampling result to obtain an initialized Gaussian field; the loss between the current image frame and the RGB color image is calculated, parameters of Gaussian primitives of all the RGB image frames in the information are iteratively optimized in sequence according to the loss, all the optimized Gaussian primitives are combined, and a reconstructed three-dimensional scene is obtained. According to the invention, the spatial consistency and accuracy of reconstruction results are improved.

The invention discloses a 6D pose estimation method and system based on an RGBD camera, and relates to the technical field of computer vision processing. Calibrating camera parameters based on the calibrated camera imaging; reconstructing a three-dimensional model and constructing a data set; segmenting the target object and calculating the actual size based on the three-dimensional model and the data set; establishing a 6D pose estimation network model based on the three-dimensional model and the data set; and performing pose estimation based on the actual size. The method has the advantages that the accuracy of 6D pose estimation is remarkably improved by combining the RGB image, the depth image and the point cloud data, object region segmentation is performed by using the SAM model, the actual size of the object is visually calculated by combining the depth information and the point cloud data, the precision of size estimation is improved, and the method is suitable for popularization and application. The introduction of depth information enables the pose estimation system to have better adaptability to objects with different sizes, shapes and surface features, and improves the robustness and accuracy of the system.

The invention discloses a partial discharge mode recognition method based on a residual neural network and an attention module, and relates to the technical field of partial discharge signal processing and recognition. The method comprises the following steps: acquiring partial discharge signal data to be detected; carrying out multi-scale denoising processing on the partial discharge signal by an improved ensemble empirical mode decomposition adaptive noise algorithm and wavelet packet decomposition combined method so as to remove noise interference and reserve main components of the signal; the time domain, the frequency domain and the phase information of the denoised partial discharge signals are converted into grey-scale maps through a Grubby angle field algorithm, a Grubby angle difference field algorithm and a Markov transfer field algorithm, and the grey-scale maps are fused to generate a feature RGB image. Inputting the RGB image into a residual neural network combined with a convolutional block attention module for feature extraction and classification; and finally, outputting a partial discharge mode identification result for fault detection and early warning of power equipment. According to the method, multi-scale denoising processing and multi-dimensional feature fusion are utilized, and depth feature extraction capabilities of the residual network and the attention module are combined, so that the recognition accuracy and robustness of the partial discharge mode are remarkably improved, and the method can be widely applied to fault diagnosis and early warning of power equipment.

The invention relates to the technical field of intelligent agricultural equipment, and discloses an automatic agaricus bisporus picking method, device and equipment based on artificial intelligence and a medium. The method comprises the steps that RGB images and three-dimensional point cloud data of the agaricus bisporus growth environment are collected in real time through a binocular vision camera and a depth sensor which are installed at the tail end of a picking robot, and environment illumination parameters and cultivation bed coordinate information are obtained; inputting the RGB image into a pre-trained lightweight convolutional neural network, identifying a pileus contour, a stipe position and a maturity level of the agaricus bisporus, calculating a space coordinate, a height and a growth inclination angle of the agaricus bisporus based on the three-dimensional point cloud data, and constructing an agaricus bisporus target database; a mechanical arm picking path is generated based on the agaricus bisporus target database, and path weight parameters are optimized online through a preset reinforcement learning model; according to the maturity grade and morphological characteristics of the agaricus bisporus, the clamping force and the rotating angle of the end effector are dynamically adjusted, and automatic picking of the agaricus bisporus is completed.

An image-guided diffusion network has two Convolution Neural Networks (CNNs). A RGB image and an IR image are concatenated with a Gaussian noise image and input to a denoising neural network that merges information from the RGB and IR images as noise is removed over many iterations. Then an enhancement neural network up-samples for Super Resolution (SR) and convolutes to generate a condition vector that controls Global Feature Modulation (GFM) at three convolution layers to generate a SRGFM enhanced fusion image. Timesteps are embedded using adaptive group normalization blocks within Adaptive Bottleneck Residual (ABR) blocks in the denoising network, which is a UNet having many levels of ABRs, and in the enhancement network before feature modulation. Global image features are detected by triple convoluting the image input to the enhancement network to generate the condition vector that controls feature modulation blocks at three layers of convolution.