362 results about "Bi modal" patented technology

The invention discloses a fine-grained multi-modal prompt learning method based on a visual language pre-training model, and relates to the technical field of visual language pre-training models.The method comprises the steps that a multi-modal prompt learning vector used for learning visual features and text features is constructed, and a fine-grained multi-modal prompt learning model is constructed based on the fine-grained multi-modal prompt learning model; global differences, subtle distinguishing visual features and category difference text features among different visual categories are generated, and a multi-modal prompt learning vector is optimized based on a cosine similarity loss function. According to the invention, through combination of dual-granularity visual prompt and text prompt, the problems that in an existing prompt learning tuning method, cooperative reasoning cannot be carried out in a dual-mode prompt representation space, and fine distinguishing features are difficult to capture are effectively solved; and superior migration generalization ability and few-sample learning ability are verified on a plurality of main image identification reference data sets, and the performance is remarkably improved in downstream tasks.

The invention discloses an unmanned aerial vehicle multi-modal feature fusion target tracking method and system based on natural language description, belongs to the technical field of computer vision and image processing, and solves the problem that in the prior art, when the quality of an image collected by an unmanned aerial vehicle is poor or image features are not obvious, the target tracking capability and the long-time tracking capability are poor. Natural language description is carried out on a traffic accident scene in an image of an unmanned aerial vehicle visual angle, and a language prompt is obtained; constructing a scene-context feature pyramid network to perform context information enhancement processing on the image of the view angle of the unmanned aerial vehicle to obtain a feature-enhanced image; respectively carrying out visual coding and language coding on the enhanced image and language prompt to obtain visual features and language feature vectors, and carrying out visual-language bimodal feature local alignment; and fully fusing the obtained aligned new language features with the visual features to obtain multi-modal features for target tracking. The method is used for multi-modal feature fusion target tracking of the unmanned aerial vehicle.

The invention relates to the technical field of information retrieval and generation, in particular to a retrieval enhancement method based on multi-modal data fusion and modal perception. According to the method, firstly, a dual-channel architecture is adopted to perform feature extraction and coding on a text and an image respectively, and mutually independent embedded representation spaces are constructed, so that high-quality collaboration and matching of cross-modal representation are realized; and a pseudo-pairing generation mechanism is introduced to effectively mine and reconstruct the existing non-paired data in the knowledge base. And designing a query modal perception and dynamic weighting mechanism for accurately controlling the fusion proportion of the image-text bimodal information in the retrieval stage so as to match the modal demand difference of different query contents. And further executing aggregation retrieval and reordering of the cross-modal information by using dynamic weighted fusion retrieval to generate a candidate set of multi-modal responses. According to the method, accurate matching and dynamic weight adjustment of the image-text content are realized, and the accuracy and expression integrity of the generated content are improved.

The invention provides a multi-mode ultrasonic fusion pressure vessel weld defect nondestructive testing method and system, and relates to the technical field of nondestructive testing. According to the method, geometric parameters of a welding seam are obtained through three-dimensional laser scanning, and an optimal scanning parameter set is generated; driving ultrasonic phased array equipment to scan for one time and synchronously acquire shear wave full-matrix capture and longitudinal wave linear scanning data; performing energy flow angular spectrum analysis and envelope analysis on the bimodal data, extracting defect feature parameters and constructing a three-dimensional feature tensor; carrying out multi-dimensional feature fusion by adopting Tucker decomposition, and enhancing a core tensor through physical modeling; generating three types of defect indication diagrams including a defect existence possibility diagram, a defect relative scale diagram and a defect space orientation diagram from the enhanced feature tensor; and the three types of indication diagrams are visually presented for comprehensive interpretation of detection personnel. Through multi-modal data fusion and physical modeling enhancement, the defect identification accuracy and detection efficiency are remarkably improved, the false alarm rate is reduced, and reliable technical support is provided for pressure vessel welding seam safety detection.

The invention discloses a medical image analysis method and system based on multi-modal image fusion, and relates to the technical field of medical image processing, and the medical image analysis method comprises the steps: generating a training data set; the training data set is composed of a medical image RGB domain image and a RAW domain image pair which are subjected to pixel-level alignment and labeling; constructing a target detection network; the target detection network comprises a backbone network and a detection head, a bimodal feature enhancement module is introduced between the backbone network and the detection head to realize deep fusion of cross-modal features, and features of an RGB domain image and an RAW domain image are respectively enhanced in semantic and spatial dimensions by using a dual enhancement mechanism; training the target detection network by using the training data set to obtain a medical image detection model; and analyzing a medical image acquired in real time by using the medical image detection model so as to quickly and accurately realize detection and classification of a lesion area in the medical image.

The invention discloses an aviation transparent part defect collection and detection method based on mixed illumination, and the method comprises the steps: sequentially employing forward illumination and backlight illumination for the same defect region of an aviation transparent part, i.e., when one illumination is carried out, turning off the other illumination, and collecting an image pair; marking and enhancing data of the obtained images of different defect areas, constructing an aviation transparent part defect detection bimodal image data set which covers defect types including contusion, scratches, cracks and point defects, and strictly aligning two modal images in the data set in space; a defect detection model is constructed, the aviation transparent part defect detection bimodal image data set is used for training, and defect detection is carried out based on an image pair; according to the invention, forward illumination and backlight illumination are adopted to shoot the same defect position, so that more accurate defect detection can be realized.

The invention relates to the technical field of steel surface detection, in particular to a bimodal target detection algorithm and system based on steel surface defects, and the algorithm comprises the steps: collecting steel surface data, and carrying out the processing to obtain a laser reflection intensity image representing two-dimensional texture information and a surface depth image representing three-dimensional depth information; and constructing a bimodal feature extraction and fusion network model: introducing deformable convolution into a YOLOv8 backbone network, constructing a parallel feature extraction strategy to realize feature fusion, fusing bimodal image features after feature extraction, and meanwhile, using an RGB image and a depth image as input of the network, so as to obtain a bimodal feature extraction and fusion network model. Optimizing a model training process by adopting a PloU loss function and a CloU loss function which are used for performing exponentiation on the IoU, and identifying an output result; according to the method, the detection rate of steel surface defects is greatly improved, the yield of products can be effectively guaranteed, and meanwhile the influence of external environmental factors on an algorithm is reduced.

The invention discloses a water surface target detection method based on bimodal image feature fusion, and relates to the technical field of water surface target detection, and the method comprises the steps: constructing a detection network based on bimodal feature enhancement, a backbone network of an infrared image input end of the detection network and a backbone network of a visible light image input end of the detection network are the same as YOLOv5 in structure, when the backbone network of the infrared image input end and the backbone network of the visible light image input end carry out feature transmission, feature fusion of infrared light and visible light is completed through a fusion enhancement module; the fused features are respectively transmitted back to the backbone network of the infrared image input end and the backbone network of the visible light image input end for further feature extraction; and detecting a water surface target by the trained detection network. According to the method, bimodal feature extraction is completed by expanding a backbone network of YOLOv5, a fusion enhancement module is introduced, and visible light and infrared feature fusion is completed in the feature extraction process.

The invention discloses a health examination data analysis method and system based on artificial intelligence, and the method comprises the steps: receiving multi-modal health examination original data of a user, and generating a multi-modal feature matrix fused with time-space correlation; inputting the multi-modal feature matrix into a lightweight dual-channel network to obtain a decoupled dual-modal feature group; constructing a hidden Markov chain based on the bimodal feature group, and generating a health state transition trajectory diagram with a timestamp; inputting the health state transition trajectory diagram into a discriminator of the generative adversarial network, and outputting a confidence score and a pathology trigger threshold of a high-risk node; and activating a rule engine according to a pathological trigger threshold, and generating a hierarchical health intervention instruction set in combination with individual living habit data. By using the embodiment of the invention, the physical examination data can be efficiently analyzed, and the accuracy of disease early warning and health intervention is optimized.

The invention provides a dual-mode image fusion method and device based on tuple disturbance and a storage medium, and relates to the technical field of image fusion processing. The method comprises the following steps: constructing a model comprising an encoder, a feature fusion network and a decoder by collecting a cross-modal image group; an encoder is trained based on a tuple disturbance comparison learning framework, cross-modal sharing and complementary features can be mined, learnable positive and negative sample pairs are constructed through a tuple disturbance module, dependence on the same mode is avoided, and generalization is enhanced; the feature fusion network performs complementary feature fusion and multi-scale semantic aggregation training by using cross-modal deep features to improve the quality of a fused image; multi-scale semantic aggregation solves the problem of detail loss, and spatial feature calibration strengthens details, so that a fused image retains bimodal characteristics, and semantic consistency is enhanced; and a combined loss function is constructed, a pixel intensity and gradient structure difference optimization model is integrated, the performance and robustness are improved, key information of the fused image is ensured, and the visual effect and the accuracy are better.

The invention relates to the technical field of audio and video processing, and provides an audio and video object intelligent tracking optimization method and system combined with deep learning. The method comprises the following steps: performing cross-modal feature collaborative extraction on an audio stream and a video frame sequence by acquiring a synchronous audio and video data group, and generating a multi-modal feature set containing audio time domain dynamic features and video space structure features; inputting the multi-modal feature set into a pre-trained association enhancement network to generate a cross-modal semantic aligned association feature sequence; constructing a tracking stability evaluation model based on the associated feature sequence, and outputting a stability index; tracking parameters are dynamically adjusted according to the stability index, an initial tracking result is calibrated, and an optimized tracking trajectory is output. Therefore, the precision and stability of object tracking in a complex scene are improved by deeply fusing the dual-mode characteristics of the audio and the video, mining the internal association between the modes and combining a dynamic evaluation and calibration mechanism.

The invention relates to the technical field of intelligent emotion recognition, in particular to a multi-modal emotion analysis method based on cross-modal attention, and the method comprises the steps: obtaining a to-be-analyzed video segment; the to-be-analyzed video segment is input into a preset multi-mode sentiment analysis model, a sentiment analysis result is output, and the multi-mode sentiment analysis model comprises an input layer used for extracting features of the to-be-analyzed video segment; the single-mode feature extraction layer is used for performing single-mode context information extraction on the features to obtain single-mode context features; the cross-modal interaction layer is used for performing cross-modal feature interaction on the single-modal context features to obtain a plurality of bimodal joint features; the multi-modal fusion layer is used for carrying out fusion processing on the bimodal joint features to obtain cross-modal interaction features; and the output layer is used for splicing the single-modal context features and the cross-modal interaction features, obtaining multi-modal fusion features and outputting an emotion analysis result. According to the invention, the emotion tendency of the speaker can be reflected more accurately.

The invention belongs to the technical field of artificial intelligence, and particularly relates to a service-oriented robot-oriented multi-modal emotion recognition method and system, and the method comprises the steps: collecting audio and video stream data of emotion changes of a user, and separating visual and voice data; extracting visual and voice emotion features through a pre-training model, and calculating prediction probability distribution of each mode; constructing a bimodal confidence quantitative model based on the distribution to obtain each modal confidence; and fusing the features by adopting a sectional type dynamic weight distribution strategy so as to identify the emotional state of the user. Visual and voice modes are fused, feature alignment is realized in combination with dynamic time warping, spatial optimization performance is shared and expressed through a confidence model, a dynamic weight strategy and a cross-modal time sequence cooperation module, and the method has high recognition accuracy, high robustness and real-time processing capacity in a complex environment and is suitable for various service scenes.

The invention discloses a knowledge distillation method for target detection. The knowledge distillation method comprises two major designs including a dynamic teacher-student mutual learning mechanism and interpretable feature decoupling. The RGB detector is used as a teacher guidance event detector to learn static semantic knowledge in a conventional illumination scene, and the event detector is used as a teacher guidance RGB detector to learn dynamic robust features in an unconventional illumination scene, so that dual-mode collaborative optimization is realized; and the original teacher and student feature space is decoupled into three parts of mode universality, mode specificity and mode irrelevance, and effective knowledge migration based on the mode universality feature is carried out between the two modes by constructing bidirectional distillation loss. The objective of the invention is to enable an RGB detector and an event detector to have a more accurate target recognition effect.

The invention discloses a shield muck volume calculation method and device based on machine vision, and relates to the field of image processing, and the method comprises the steps: carrying out the feature extraction and fusion of a target depth image and a target RGB image obtained in real time through an image segmentation network based on bimodal input, and obtaining a target depth image and a target RGB image; performing image segmentation on the target RGB image according to the obtained fusion features to obtain a residue soil mask image of the target RGB image and coordinates of a residue soil bounding box; according to the coordinates of the muck bounding box and the muck mask image, the volume of the shield muck is calculated through a volume calculation method; according to the method, the accuracy of volume calculation of the shield muck is improved on the basis of good real-time performance, and the problem that the accuracy of the volume of the shield muck calculated by an existing non-contact measurement technology of the volume of the shield muck is low under the high real-time performance requirement is solved.

The invention relates to the technical field of vehicles, in particular to an intelligent cabin man-machine cooperative control method and device, a vehicle, a medium and a product. Performing fusion processing on the voice signal data, the visual signal data and the tactile signal data of the current driver by using a preset dynamic weight distribution model to obtain a multi-modal fusion control instruction; and under the condition that the multi-mode fusion control instruction meets a preset sight-voice dual-mode condition, synchronously issuing the multi-mode fusion control instruction to a plurality of target execution devices in the intelligent cabin so as to drive each target execution device to perform cooperative control. Therefore, by constructing a cooperative control mechanism of multi-mode sensing and dynamic weight distribution, the problems that the false wake-up rate is high, the voice and graphical interface cooperative efficiency is low and the interaction mode cannot adapt to a complex driving environment due to a single or separated interaction mode in the related technology are effectively solved.

The invention discloses a method and a device for filling and quantifying multi-source data of a power system based on a bimodal hybrid algorithm in extreme weather, which belong to the technical field of power distribution networks and are based on a reverse chaos whale transfer algorithm and combined with meteorological characteristics to dynamically adjust and construct a load prediction model to fill missing data and output distribution parameters and weight coefficients. Meanwhile, a meteorological condition matrix is introduced, a Gaussian mixture model is constructed, and the uncertainty of data is quantized and filled through a Monte Carlo subset deduction algorithm. The method effectively solves the problems that a traditional method is low in processing efficiency and large in error in extreme weather, experiments show that the method is high in convergence speed and high in estimation precision, the accuracy and reliability of power system data are remarkably improved, and guarantee is provided for stable operation of a power dispatching system.

The invention discloses an industrial art material detection method based on multi-modal deep learning, particularly relates to the field of material analysis, and is used for solving the problems that cross-modal data alignment of curved surface utensils in highlight and multi-layer coating scenes is difficult, and material boundaries are easy to drift along with shooting batches and visual directions. The method comprises the following steps of: positioning a hyperspectral line scanning fragment; constructing a local reference grid to realize initial pairing; generating pixel-level residual image quantization dislocation; estimating luminosity mapping to form consistent bimodal data pairs after eliminating a high-reflectivity region; iteratively supplementing anchor points or adjusting grid rigidity, generating a stable material graph, updating an index table and a consistency record, ensuring cross-batch reusability and traceability, and improving the material detection precision.

The invention provides an infrared and visible light image fusion method based on text semantic consistency guidance, which relates to the technical field of multi-modal image fusion, and comprises the following steps: respectively carrying out fine-grained text semantic generation on infrared and visible light images and mapping the infrared and visible light images to a unified embedding space; bidirectional compensation and enhancement are carried out on text semantics through a cross-modal attention mechanism, and unified text semantic priori is constructed; a structure-intensity decoupling double-branch encoder is adopted, and structure texture features of visible light and intensity significant features of infrared light are extracted respectively; under the prior guidance of text semantics, the bimodal visual features are aligned in a shared semantic space through explicit semantic consistency constraint and implicit semantic distribution consistency constraint; and finally, taking the text semantic priori as a global modulation signal, and carrying out adaptive weighted fusion and decoding on the aligned features to generate a fused image. The problems that an existing method is insufficient in semantic modeling and poor in fusion result consistency are effectively solved.

The invention relates to the technical field of cable processing, in particular to an automatic material taking mechanism for cable processing equipment, and the mechanism comprises an incoming material arraying and conveying unit which is used for flattening scattered cables and conveying the cables through a conveying belt synchronized with an encoder; the industrial vision or AI vision unit is used for performing vision detection and vision measurement on the cable in the conveying process; the material taking execution unit comprises an execution mechanism with the X-Y-theta degree of freedom and the Z degree of freedom and a bimodal flexible gripper; the posture correcting and guiding unit is used for correcting the direction of the end of the grabbed cable and standardizing the feeding posture of the grabbed cable; according to the automatic material taking mechanism for the cable processing equipment, the supplied material arraying and conveying unit is arranged to be combined with the industrial vision or AI vision unit, and the supplied material arraying and conveying unit is arranged to be combined with the industrial vision or AI vision unit; the defects that vibration stop photographing or a general taking and placing system is low in flexible cable separation efficiency and prone to missing detection are effectively overcome.

The invention provides a driving authority management method and device based on biological feature verification and a medium, and belongs to the technical field of vehicles. The method comprises the steps of collecting facial features, voiceprint data and driving behavior data by detecting a starting operation of a current driver; the identity of the current driver is recognized by using a multi-modal biological recognition algorithm, and the accuracy and anti-counterfeiting capability of identity verification are improved by adopting a bimodal fusion scheme of face recognition and voiceprint recognition; when it is determined that identity recognition of the current driver succeeds, a driving account is determined to judge whether the driver has the use permission or not, a binding relation of driver identity-account-function permission is established, and accurate matching of the intelligent driving permission and the driver qualification is achieved; the open state of the intelligent driving function is dynamically controlled based on a permission verification result, if the permission exists, the intelligent driving function is automatically prepared, if the permission does not exist, the function is locked, a clear prompt is given, and the safety risk that the intelligent driving function is used before being learned is avoided from the source.

The invention provides a class incremental learning method based on prompt guidance and multi-modal fusion, and relates to the technical field of artificial intelligence and computer vision. The method comprises the following steps: firstly, performing semantic extension on a category label, and constructing semantic enhanced text representation through a text encoder; then block embedding and hierarchical feature extraction are carried out on the input image by using a pre-trained visual encoder, a cross-modal unified embedding space is constructed, a bimodal prompt gating fusion module is introduced into the unified embedding space, and adaptive weighting is carried out on text prompt and image prompt according to gating weight to generate fusion prompt; through a bimodal prompt collaborative filtering module, screening out a prompt set most relevant to the current task according to the similarity of the semantic features of the image and the text; the pre-training backbone network is frozen in the increment stage, only prompt parameters and fusion layer weights are optimized, a joint loss function is used for parameter updating, finally, image and text data are input in the reasoning stage, cross-modal similarity is calculated, and a classification prediction result is output.

The invention discloses a vending machine multi-mode emotion perception and personalized dialogue generation method and system, and the method comprises the steps: collecting a face image and a voice signal of a user through a camera and a microphone, and outputting a dual-mode emotion tag through the analysis of a face expression and a voice emotion; fusing the label and the dialogue text to generate a structured Prompt containing emotion, intention and context; based on an Agent framework multi-modal large model, combining context history and a personalized strategy to generate a reply text matched with the emotional scene; according to the text and the comprehensive emotion label, TTS parameters are dynamically adjusted, and anthropomorphic voice is output; in multiple rounds of dialogues, styles are kept consistent through an emotion smoothing formula, and emotion-verbal skill-conversion data are precipitated in combination with a user feedback optimization strategy. The interaction limitation of a traditional vending machine is broken through, multi-mode emotion perception, personalized personification interaction and continuous evolution are achieved, the vending machine is promoted to be upgraded to an emotional shopping guide platform, user experience and sales transformation are improved, and a technical model is provided for intelligent retail.

The invention discloses a high-precision face recognition method and system which are suitable for complex illumination and unconstrained position scenes. The method is realized through cooperation of a multi-modal acquisition guide strategy driven by a state machine and a self-adaptive fusion mechanism based on a physical multiplier array: the state machine controls logic to generate a position-attitude sequence instruction, and drives a user to synchronously provide RGB-NIR image pairs in upper, middle and lower screen areas and left and right side face attitudes; the double-flow convolutional neural network extracts an aligned bimodal feature map through shared weight constraint; the attention fusion module dynamically generates a space-channel joint attention weight based on a local signal-to-noise ratio difference through a multiplier array and a weight register realized by hardware, and performs pixel-by-pixel weighted fusion on the bimodal features. And a physical mapping relation between state machine position change and modal weight distribution is established through coupling training. According to the invention, through algorithm hardening and hardware cooperation, the accuracy and real-time performance of cross-illumination and cross-position unconstrained face recognition are significantly improved.

The invention relates to a multi-modal survival prediction method fusing a digital pathological image and omics data. The method comprises the following steps: acquiring a full-view digital pathological image and omics data of an esophageal squamous cell carcinoma patient; carrying out preprocessing and feature extraction on the full-view digital pathological image and the omics data to respectively obtain a pathological embedding representation and an omics embedding representation; fusing the pathological embedding representation and the omics embedding representation by using a multi-head attention mechanism of Transform to obtain a preliminary fusion feature; performing deep interaction on the pathological embedding representation and the preliminary fusion feature to obtain a final fusion feature; and predicting a survival result based on the final fusion feature. According to the method, bimodal sharing of pathological images and omics data and full fusion of features are realized, and prognosis survival prediction of esophageal squamous cell carcinoma cancer patients can be better realized.

The invention discloses a method and a system for tracking and detecting an occluded target in an electric power scene based on thunder-vision integrated multi-mode spatio-temporal adversarial learning, and relates to the field of intelligent inspection of electric power equipment, and the method comprises the steps: inputting a structured point cloud matrix and a visual image into a double-branch feature extraction model, respectively extracting geometric features and semantic features by using a double-branch feature extraction model; inputting the bimodal fusion features into a dynamic weight adaptive fusion mechanism based on an adversarial training strategy to obtain a fusion weight coefficient of the target in a short-time shielding state in the power scene; and a double-buffer memory pool is constructed, and a three-level progressive trajectory prediction compensation mechanism is combined to compensate a target trajectory position in a long-time shielding state in a power scene. According to the method, the shielding state quantitative model is constructed, the fusion weight distribution proportion can be effectively and dynamically adjusted in real time according to the visual sensor, and when it is detected that the target equipment enters the shielding state, the visual data weight is automatically reduced to the safety threshold value.

The invention discloses an unmanned aerial vehicle small target detection method and system based on bimodal image fusion. The target detection method comprises the steps of obtaining a to-be-detected unmanned aerial vehicle image; the to-be-detected unmanned aerial vehicle image is input to a target detection model, a detection result is obtained, and the target detection model comprises a backbone network module, a neck network module, a detection module and an output module; the backbone network module is used for acquiring feature maps of different scales; the neck network module is used for performing different-level feature fusion on the feature map to obtain a fused feature map; the detection module is used for detecting the fused feature map to obtain a detection result; and the output module is used for outputting the detection result. The problems of low detection precision and high omission ratio caused by small target size and large modal information difference in small target detection of the unmanned aerial vehicle are solved.

The invention provides a skin lesion auxiliary detection method and system based on multi-modal information fusion, and belongs to the technical field of computer vision and application. The method comprises the following steps: firstly, acquiring and constructing bimodal data including a dermatoscope image and a text of a patient, and preprocessing the data; secondly, respectively adopting an improved convolutional neural network and a text coding model, and extracting deep and high-dimensional features from the skin disease image and the self-described text of the patient; afterwards, an improved symmetric gating cross attention fusion module is constructed, fine-grained interaction and alignment between visual and text features are realized through a bidirectional cross attention mechanism, and a gating unit is utilized to adaptively adjust a fusion weight; and finally, inputting the deeply fused multi-modal feature vector into a classifier to obtain a confidence score of each skin lesion type so as to provide reference for dermatologists. According to the method, the diagnosis accuracy, robustness and interpretability are synergistically improved, and the method has great clinical application value.

The invention discloses a cross-modal attention collaborative jail break attack method, and belongs to the technical field of artificial intelligence security. The method comprises the following steps of: constructing an input sequence representation containing a system prompt, an adversarial image representation, a malicious query and an adversarial text suffix according to a causal self-attention mechanism; inputting the sequence into a visual language model to execute forward propagation; based on the designed attention-oriented loss collaborative function, optimizing adversarial image representation through a joint gradient optimization algorithm and updating an adversarial text suffix to optimize an attack target; iteratively circulating until convergence, and outputting the optimized unified multi-modal knowledge; and finally, utilizing the knowledge to construct an attack sequence to realize jailbreak. According to the method, accurate control on an internal attention mechanism of the visual language model is realized for the first time, and through visual-text dual-mode cooperative attack, the attack success rate is remarkably improved while high concealment is kept, and the important driving force for promoting the progress of a safe alignment technology is achieved.

The invention relates to the technical field of litchi phenotype analysis, in particular to a litchi phenotype recognition and weight estimation method based on multi-modal learning, and adopts the technical scheme that an RGB (Red, Green, Blue) image and a depth image are used as bimodal input and are input into a LitchiPhenoNet model to realize detection and segmentation of fruits; an RD fusion module is adopted to significantly improve the multi-modal feature expression capability and segmentation performance, and the fusion strategy integrates texture details of an RGB image and space structure information of a depth image, so that the accuracy and robustness of detection and segmentation are improved; on the basis of geometric features and abstract intermediate features directly extracted by LitchiPhenoNet, fruit weight estimation is performed by adopting a multi-modal regression method, a regression model combines information from two aspects of vision and physics, and by combining the two types of features, the regression model can effectively improve the utilization efficiency and prediction precision of multi-modal data, so that the prediction accuracy of the fruit weight is improved. The multi-modal regression framework effectively integrates complementary information of geometric features and intermediate visual features, so that the accuracy and robustness of litchi weight estimation are improved.