414 results about "Multimodal image" patented technology

A real-time system for the multimodal generation of virtual reality scenes based on artificial intelligence for the creation of immersive three-dimensional environments from natural language narratives, consisting of: a speech capture module configured to continuously record a user's spoken narrative via one or more directional microphones, preprocesses the captured signal by noise reduction and temporal alignment, and outputs a digital speech stream; A speech-to-text processing unit that is operationally coupled to the speech capture module and configured for real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining contextual continuity throughout the evolving narrative. a semantic interpretation processing unit that is communicatively linked to the speech recognition unit and configured to perform natural language understanding techniques to extract contextual entities, spatial references, temporal relationships, and object attributes from the transcribed narrative; the engine includes a large language model that is fine-tuned for spatial reasoning tasks; a scene graph generation module configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; a multimodal image-language model processor coupled with the scene graph generation module, wherein the processor is configured to retrieve, adapt, or synthesize appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and align these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller configured to create a cohesive virtual scene from the aligned assets, perform real-time rendering using a GPU-accelerated ray tracing pipeline, and produce a stereoscopic visual output that corresponds to the evolving narrative; A head-mounted virtual reality visualization device connected to the rendering engine and configured to display the generated immersive environment to the user in real time. The device features motion sensors and inside-out tracking cameras to detect head and body movements, dynamically updating viewing angles and perspective within the rendered scene; and a bidirectional feedback module integrated into the head-mounted device and connected to the semantic interpretation processing unit; the module is configured to interpret corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization; The system continuously updates the virtual scene as the narrative develops, ensuring temporal synchronization between speech input and rendered output below a defined latency threshold, thus enabling a natural, dialogic construction of complex three-dimensional virtual environments.

The invention relates to the technical field of underwater fish school monitoring, and discloses an underwater fish school monitoring statistical system based on image fusion. According to the system, underwater video streams and sonar reflection intensity data of different spectral bands are acquired through an underwater multi-source image acquisition module, and time-space synchronous multi-modal image data streams are generated through timestamp alignment; a fish school contour reconstruction module is used for segmenting a fish school contour boundary and fusing visible light texture and sonar geometric features to generate an underwater three-dimensional fish school distribution set; the dynamic track mapping module tracks the mass center displacement, calculates the movement rate and the direction deviation angle, and correlates the water area depth to generate a dynamic track topological graph; the behavior anomaly analysis module extracts environment data based on the track mutation node, and detects aggregation density change and direction dispersion to mark an anomaly feature cluster; and the population statistics output module integrates the data, performs classified statistics on population distribution, a quantity threshold value and a migration path overlap ratio, and finally generates a fish school quantity distribution statistics thermodynamic map.

The invention relates to the technical field of visual positioning, and provides a visual positioning method and system for precise connector assembly. Performing multi-mode HDR fusion and distortion correction on the original connector image to obtain a multi-layer fusion image matrix; and performing feature region coarse positioning in combination with a composite convolutional neural network to obtain a connector coordinate set, performing geometric feature fine positioning according to the connector coordinate set and the multilayer fusion image matrix to obtain a key point coordinate set, and performing hand-eye calibration and dynamic compensation on the key point coordinate set to generate a pose instruction. And performing vision-force control hybrid assembly processing on the pose instruction according to the sensor feedback data to obtain assembly completion state data. Through multi-modal image fusion, deep learning coarse positioning, precise geometric registration and dynamic cooperation of vision and force control, the speed, precision and stability of precise connector assembly are improved.

The invention provides a robot electrical equipment defect detection system based on multi-modal image processing. The system improves the accuracy and reliability of electrical equipment defect identification. Infrared and visible light images are jointly collected, and through a registration algorithm of multi-source features and equipment structure priori, space-time alignment of multi-modal images is achieved. Then, a dynamic weighted fusion strategy is utilized to generate fusion features with higher discriminative ability, and abnormal features are extracted through a double-branch mechanism to be verified with thermophysical consistency; and finally, constructing a neural network model fused with physical prior, and performing defect classification and positioning output on the verified feature data. According to the method, the structure and thermal information are fused, a physical constraint mechanism and a joint training strategy are introduced, the robustness and engineering interpretability of the system under complex working conditions are remarkably improved, and the method has a wide application prospect.

The invention belongs to the field of image processing, and provides a gynecological tumor image processing method and system based on AI multi-modal image analysis, and the method comprises the steps: 1, obtaining an original image of a patient, and obtaining a structure mask and an image frame sequence after period alignment and structure normalization based on the original image; step 2, obtaining a focus mask sequence after structure limitation based on the image frame sequence; step 3, respectively acquiring a modal structure semantic tensor of each image in the image frame sequence, and acquiring a fused semantic feature tensor based on the modal structure semantic tensor; 4, obtaining a final focus mask based on the fused semantic feature tensor and the structure mask; and step 5, obtaining a response visualization graph based on the focus mask. The method is clear in technical structure, coherent in task chain and independent in model interface, has real deployment and continuous evolution capabilities, and is particularly suitable for gynecological image AI auxiliary system scenes under periodic driving.

The invention relates to the technical field of tumor interventional operations, and discloses a visual navigation method based on a tumor interventional operation robot. The method comprises the following steps: acquiring real-time medical image data of a tumor area containing multi-modal imaging information so as to comprehensively present anatomical details; and performing three-dimensional reconstruction on the image data to generate a tumor area three-dimensional anatomical structure model capable of visually displaying a space structure. Key anatomical feature points are extracted based on the model, space coordinates are calculated, a surgical robot intervention path is planned according to the coordinates, and an initial navigation track is generated; and continuously collecting real-time pose data of the robot in an operation, dynamically matching the real-time pose data with the initial navigation trajectory, adjusting motion parameters according to a matching result, and generating a corrected navigation instruction. The method can reflect the intraoperative anatomy condition in real time, dynamically optimize the path, solve the problems that traditional navigation depends on preoperative static images and lacks real-time adjustment, reduce operative complications and improve the treatment effect of patients.

The invention discloses a detection method and device for intelligent visual detection of parts and a storage medium, belongs to the technical field of industrial quality detection, and aims to solve the problems that surface and internal defects of complex parts are difficult to identify synchronously and the identification precision is low. The method comprises the following steps: acquiring multi-modal image data obtained through structured light three-dimensional imaging and laser ultrasonic scanning; performing geometric registration and scale normalization on the image to generate a fused image; carrying out image preprocessing and edge analysis, and extracting a region of interest; dividing the candidate region of interest into a plurality of image blocks, and inputting the image blocks into an anomaly detection network; calculating a reconstruction error between the original image block and the reconstructed image block, and generating an abnormal scoring graph; and finally, extracting a defect area through image post-processing, and outputting information such as a defect type, a spatial position, a geometric dimension and a severity level. According to the method, the robustness and accuracy of multi-modal defect identification are improved, and the method is suitable for an online visual inspection task of an industrial production line.

The invention discloses a tumor space-occupying brain network neural image alignment method based on multi-modal fusion, and belongs to the technical field of medical image processing and artificial intelligence crossing. The method comprises the following core steps of multi-modal image heterogeneous feature decoupling, tumor occupation deformation field modeling, functional network topological structure maintenance, cross-modal feature adversarial alignment, dynamic deformation constraint optimization and clinical interpretability verification, and construction of a three-dimensional non-rigid registration network based on a double attention mechanism. And differential homeomorphic mapping of a tumor focus area and normal brain tissue is realized through the cascaded spatial transformation module. Aiming at the problems of insufficient multi-modal feature alignment and brain network topology distortion in the prior art, the invention provides a function connection constrained cross-modal fusion strategy, a graph convolution network is adopted to encode resting state function connection features, and network node displacement caused by tumor occupation is dynamically corrected in combination with deformable convolution and a bidirectional feature competition mechanism; a space consistency loss function based on white matter fiber bundle tracing is designed, and through diffusion tensor imaging feature guide structure-function bimodal joint optimization, the problems of insufficient registration precision in a focus area and whole brain network connection distortion of a traditional method are solved. Experiments show that the registration precision of the method in glioma cases reaches 0.82 mm and is improved by 37% compared with that of a traditional method, and dissection-function consistency of functional network reconstruction around tumors is remarkably improved.

The invention discloses a deep learning-based multimodal image fusion method for soft tissue photoacoustic / ultrasound imaging. Steps: an ultrasound-photoacoustic imaging device acquires photoacoustic and ultrasound images of human soft tissue and performs size normalization processing; an input spatial transformation module converts the images to the YCbCr space; an input pre-convolution module modifies the number of data channels; an input multi-scale feature extraction module extracts salient features from the source images; an input filter prediction module derives multi-scale filters; and an input filter fusion and adaptive enhancement module combines the input source images to obtain the final fused result. The invention has superior fusion performance compared to several traditional fusion methods and deep learning-based fusion methods, and more importantly, it exhibits excellent real-time performance. Furthermore, various modes of photoacoustic / ultrasound fusion extension experiments have verified the effectiveness of the method proposed in the invention.

The invention discloses a ground target identification method and system based on multi-modal image feature fusion, and the method specifically comprises the steps: firstly obtaining an infrared image, a visible light image and an SAR image of a ground target, carrying out the image amplification of the images of three modals, and adding classification labels; performing network training by adopting different neural network models to realize preliminary classification of targets; thirdly, removing classification heads from the three pre-trained neural network models, fixing feature extraction network parameters, extracting features of different modal data by using a pre-trained feature extraction module, and performing feature fusion by using a channel-space-modal three-level attention fusion module; performing neural network model training again by using the multi-modal image and the label to obtain a trained neural network model; and finally, inputting a to-be-recognized image into the trained neural network model for target recognition. According to the method, the precision of ground target identification in a complex environment is improved, the calculation overhead is low, the adaptability is high, and the expansibility is high.

The invention discloses a target identification tracking method based on modal feature completion, a training method and system of an information fusion network model, a storage medium, equipment and a computer program product. Determining missing fusion feature data of the multi-modal image data; inputting the missing fusion feature data into an information fusion network model to obtain complete fusion feature data of the multi-modal image data; and inputting the complete fusion feature data into the target image processing model to determine an image processing result of the multi-modal image data. According to the method, the missing modal features are complemented through the information fusion network model, complementary information of different modals is effectively integrated, the stability, robustness and reasoning effect of the system are improved, and reliable target recognition and tracking results can still be obtained in a complex environment.

The invention relates to a cloud-edge collaborative medical image intelligent diagnosis method and device, equipment and a medium. The method comprises the following steps: firstly, obtaining original image data, and carrying out calibration processing on the original image data through a preset multi-modal image standardization model to obtain a standard image data set; performing feature extraction and grading processing on the standard image data set at an edge end to obtain a grading feature set, transmitting the grading feature set to a cloud end through a bandwidth allocation strategy, and generating a cloud end receiving feature subset; constructing a feature expression matrix based on the cloud receiving feature subset, using a network topology structure to carry out graph analysis, and extracting a path to generate a diagnosis path set; and finally, feedback diagnosis information is formed based on the diagnosis path set, and a task allocation optimization result is obtained by dynamically adjusting a cloud side task allocation proportion. According to the method, efficient processing and diagnosis optimization of medical image data are realized, efficient collaboration of cloud edge resources is ensured, timeliness and accuracy of medical image diagnosis are effectively improved, and resource scheduling requirements in different scenes are met at the same time.

The invention discloses a diabetic foot early-stage detection method combining infrared and visible light imaging, and relates to the technical field of diabetic foot medical imaging diagnos.The diabetic foot early-stage detection method comprises the steps that multi-mode image collection and standardization processing are conducted, infrared images and visible light images are synchronously collected, and standardization processing such as size normalization is conducted; image registration and space alignment are carried out, marking points are set based on foot anatomical features, and feature extraction, mismatching point elimination and transformation matrix calculation are carried out; extracting and screening multi-dimensional features, extracting temperature and structural features, and screening by using a Relief-F algorithm; feature lesion recognition and classification are fused, and lesion probability is output through a double-branch convolutional neural network; carrying out detection result verification and feedback optimization, and comparing a clinical diagnosis optimization model; and generating a detection report and storing data, and generating a report containing the fused image. The early lesion detection precision is improved through multi-modal fusion, individual and environment differences are adjusted and adapted in a personalized mode, and reliable technical support is provided for clinic.

The invention relates to a multi-modal image matching method and system based on learning features and epipolar geometric constraints. The method comprises the following steps: carrying out edge enhancement processing on an input image through wavelet transform; extracting a multi-scale dense feature map based on the transformed convolutional neural network, and generating a feature descriptor with rotation and scale invariance in combination with principal direction normalization; adopting an FLANN algorithm and dynamic distance constraint to realize preliminary feature matching; and introducing a basic matrix construction and epipolar geometric consistency verification mechanism, and eliminating mismatching point pairs in combination with an RANSAC affine constraint model. According to the method, image enhancement, deep learning and geometric verification strategies are fused, the problems of radiation nonlinearity and geometric distortion caused by imaging mechanism differences among multi-modal images are effectively solved, the matching precision and robustness are improved, and the method is suitable for remote sensing application scenes such as optical-SAR registration, multi-source image fusion and earth surface change detection.

The invention relates to the technical field of industrial visual inspection, in particular to a cigarette packet surface defect detection method and system and a storage medium, and the method comprises the following steps: optimizing a light field by adopting a multi-mode light field regulation and control model; a distributed camera array is arranged, and a multi-angle image, a multi-mode image and a multi-polarization-angle multi-spectral image of the surface of a to-be-detected cigarette packet are obtained; according to the multi-angle image and the multi-modal image, obtaining a three-dimensional reconstruction depth map by adopting a depth estimation model; based on the multi-polarization-angle multispectral image, defect detection is carried out, and a defect edge is extracted; constructing a defect classification model based on self-supervised learning; carrying out joint training on the multi-modal light field regulation and control model, the depth estimation model and the defect detection and defect classification model to obtain a joint detection model; and performing defect detection on a to-be-detected cigarette packet by adopting the trained joint detection model. According to the embodiment of the invention, by fusing the dynamic adjustment light field, the deep learning algorithm and the multi-angle imaging, the defect detection performance in the high-reflection environment is optimized.

The application discloses a memory reinforcement deep unfolding multi-modal image fusion method with downstream task enhancement, which comprises the following steps: collecting infrared images and visible light images, and dividing training set and test set; establishing an optimization target and solving, to obtain an iterative formula; using a neural network to replace a proximal operator in the iterative formula, to obtain a neural network structure; sending the training set into the neural network to obtain a fusion picture, and calculating a total loss according to the fusion picture, the infrared image and the visible light image; updating parameters of the neural network according to the total loss to obtain an updated neural network; inputting the infrared image and the visible light image of the test set into the updated neural network to obtain a fusion image. The application makes the fused image have characteristics easy to be distinguished by a downstream task network, and can realize the best performance on a data set, while performance and interpretability are taken into account, and the application has rationality and applicability.

The invention discloses an unmanned aerial vehicle detection and tracking method based on a multi-modal image, and belongs to the field of target detection and tracking. The unmanned aerial vehicle in the air is detected through the combined action of multiple modes, so that the reliability of unmanned aerial vehicle detection in various environments is realized, and identification and classification of the unmanned aerial vehicle are realized; in the tracking stage, a prediction structure in which LSTM and a Kalman filter are fused is introduced, so that the state modeling capability under the conditions of nonlinear motion and shielding of a target is enhanced, and the stability and continuity of tracking are improved; a fusion multi-stage matching strategy is adopted to effectively improve the matching accuracy and robustness in a complex environment; meanwhile, a confidence shunt strategy and a trajectory management mechanism are combined, so that the response speed to a new target and the fault-tolerant capability to a lost target are improved; through combination of global motion estimation and height information judgment, accurate correction and target distinguishing of the track of the unmanned aerial vehicle are realized, aliasing and misjudgment are avoided, and the detection and tracking capability of the unmanned aerial vehicle is remarkably improved.

The invention discloses a breast surgery puncture positioning system, which relates to the technical field of image positioning, and is characterized in that a preoperative medical image and an intraoperative ultrasonic image of a patient are acquired to generate a three-dimensional model, an optimal puncture path is planned by receiving a needle insertion point of a user, and the spatial pose of a puncture needle is tracked; the optimal puncture path and the position and posture of the puncture needle are displayed on the real-time ultrasonic image in an overlapped mode, real-time deviation is calculated, the puncture needle is guided and adjusted to the target focus, and real-time compensation is carried out based on the respiratory signal and / or the displacement of the reference datum. Through the multi-modal image fusion and real-time dynamic tracking technology, the puncture precision and hit rate are remarkably improved, meanwhile, the system intelligently plans a safe path and provides an AR navigation interface, the operation difficulty and dependence on doctor experience are greatly reduced, physiological displacement interference is overcome through respiratory gating and a motion compensation mechanism, and the system has the advantages of being high in accuracy and high in reliability. Accuracy and reliability in the whole operation process are guaranteed, and finally standardized and safe mammary gland puncture diagnosis and treatment operation is achieved.

The invention discloses an incineration condition multi-scale identification method based on a capsule network. The method comprises the following steps: S1, collecting multi-modal image sequence data of a waste incineration hearth; s2, obtaining a weighted multi-scale SE attention feature map; s3, outputting a time sequence stable waste incineration hearth working condition judgment vector; s4, a garbage incineration hearth working condition classification result is obtained; s5, updating the combustion condition identification model; and S6, the garbage incineration hearth working condition classification result is input into an edge control interface, and a control signal for the combustion process of the incinerator is generated. According to the method, the characteristic weight of each scale channel can be adaptively adjusted according to the actual working condition of severe change of the temperature field of the incineration site and smoke disturbance, so that the high robustness and fine granularity discrimination capability of the working condition identification characteristics are still kept under the complex conditions of high smoke shielding, high moisture garbage and large piece feeding; actual measurement working condition identification accuracy can be kept above 95% for a long time.

The invention discloses a multi-mode head and neck tumor segmentation and survival prognosis prediction method. The method comprises the following steps: constructing a DE-LS-UNet model; training the DE-LS-UNet model to obtain a trained DE-LS-UNet model, and inputting the new PET image and the CT image into the trained DE-LS-UNet model to obtain a predicted tumor segmentation mask; extracting multi-modal radiomics features according to the predicted tumor segmentation mask, and fusing clinical features through the multi-modal radiomics features to generate multi-source features of the patient; inputting the multi-source features of the patient into the survival analysis model for training to obtain a trained survival analysis model, and inputting the features of the patient into the trained survival analysis model to obtain a prognosis prediction result. According to the method, a feature extraction and fusion mechanism is provided, and a survival analysis model strategy is combined, so that the stability and prediction performance of survival prognosis modeling can be enhanced while the image segmentation precision is improved.

The invention provides an analysis method and system based on a data model in the field of the gastroenterology department, and the method comprises the steps: obtaining endoscope video stream data, obtaining a clean image through the endoscope video stream data, carrying out the fluid reflection inhibition, obtaining a reflection-removed image, synchronously obtaining a CT enhanced scanning sequence and an ultrasonic elastogram, and carrying out the feature extraction. The method comprises the following steps: acquiring an endoscope anchor point, a CT anchor point and an ultrasonic anchor point, performing Lie group optimization registration, acquiring an optimal registration matrix, acquiring multi-modal image data, performing cross-modal data registration, improving the precision of multi-modal data registration, acquiring lesion state evolution data through the optimal registration matrix, and inputting the lesion state evolution data into a neural differential equation, so as to improve the accuracy of multi-modal data registration. A long-term evolution track is obtained, historical treatment records are synchronously obtained, a decision report is generated, treatment is assisted through the decision report, dynamic modeling is carried out through historical image data and real-time image data of a patient, and the accuracy of patient condition data prediction is improved based on the continuity of a focus spatio-temporal evolution law.

The invention belongs to the technical field of image recognition and power grid capital construction sites, and particularly relates to a power grid capital construction dynamic risk recognition method based on a multi-modal image and LLM deduction, and the method comprises the steps: limiting the relation between an inspection object and a design based on a stress chain and an equipotential chain, and generating a constraint framework; performing multi-channel sampling on objects in the constraint framework; after the channels are aligned, a state evidence packet is generated, and the design relation and the state evidence packet are aligned; a stress chain and an equipotential chain are assembled, and a local topological graph is obtained through cutting and minimum necessary path selection; based on the local topological graph, mapping mechanism elements of the nodes and the edges into a fixed semantic slot position and action mode template, and generating mechanism description; and outputting a dynamic risk source list based on the mechanism description and the local topological graph. According to the method, the multi-modal image is subjected to constraint mapping to form the mechanism elements and the minimum necessary topology, and deduction is carried out on the stress chain and the equipotential chain, so that the dynamic risk source is accurately identified, and evidence-level traceable decision output is realized.

The invention discloses a civil engineering foundation pit deformation monitoring system based on image analysis, and relates to the technical field of foundation pit deformation monitoring, and the system collects and enhances a multi-modal image, eliminates interference, corrects image distortion, tracks key points, constructs a panoramic image, generates a point cloud, extracts deformation features, and constructs a deformation map. The method comprises the following steps: identifying a construction disturbance event, constructing a deformation chain and performing causal reasoning, identifying a high-risk area, predicting a deformation trend and outputting an early warning, displaying a monitoring result, collecting user feedback and synchronizing system data. A deformation track is accurately extracted through image correction and key point tracking, a three-dimensional deformation map is constructed in combination with point cloud modeling, causal analysis is carried out based on construction disturbance and a deformation path, a high-risk area is predicted and identified by using a time sequence trend, an early warning is given out, user feedback collection and result synchronization are supported, and monitoring intelligence and response efficiency are improved.

The invention relates to the field of medical treatment, and discloses a method for treating chronic insomnia by percutaneous magnetic stimulation of stellate ganglion block, which comprises the following steps: preprocessing: determining the initial position of stellate ganglion, tissue initial parameters and initial stimulation parameters through multi-modal image positioning and initialization to obtain reference data; dynamic sensing: collecting neck tissue displacement data, temperature field data and electromyographic signals in real time, and calculating spatial position variation of stellate ganglions and time-varying parameters of tissue electromagnetic characteristics; and modeling optimization: constructing a time-varying electromagnetic field distribution model based on the reference data, the spatial position variable quantity and the time-varying parameters. The method comprises the following steps: acquiring neck tissue displacement, a temperature field and an electromyographic signal in real time through reference data, capturing a stellate ganglion space position change and tissue electromagnetic property time-varying rule, constructing a time-varying electromagnetic field distribution model, and solving target stimulation parameters adaptive to a real-time tissue state through an optimal control algorithm.

The invention discloses a multi-modal image fusion assisted precise tumor needle biopsy guide system, and aims to solve the problem that the traditional needle biopsy guide technology is insufficient in precision. The system innovatively adopts a multi-scale feature fusion algorithm and an adaptive weight fusion strategy, deeply fuses ultrasonic, CT, MRI and other multi-modal images, and comprehensively improves the information amount of the images. Through real-time force feedback and path adjustment, in combination with VR / AR guide interaction, intelligent puncture guide is realized, and safe and accurate operation is guaranteed. Tumor features are extracted by using a deep learning algorithm, accurate tumor recognition and dynamic image analysis are realized, and the tumor growth trend is predicted. According to the system, the precision and success rate of needle biopsy are remarkably improved, the pain of a patient and the medical cost are reduced, powerful support is provided for personalized precise treatment of tumors, and the system has extremely high clinical application value.

The invention relates to the technical field of image updating, in particular to a multi-modal image real-time updating method and system for temporal bone surgery. The method comprises the following steps: acquiring a real-time multi-modal image, performing pixel-by-pixel frequency domain reconstruction optimization, and constructing a frequency spectrum enhanced fusion image; performing reverse geometric transformation compensation on the spectrum enhancement fusion image to obtain a space alignment optimization image; performing real-time visual contrast enhancement on the space alignment optimization image, and constructing a visual enhancement image; carrying out inter-frame difference calculation on the vision enhanced image, carrying out real-time increment updating optimization, and constructing an increment updating image sequence; and performing multi-level cache rendering management and parallel execution based on the incremental updating image sequence. The temporal bone surgery safety and efficiency are improved through real-time and efficient image updating.

The invention discloses an intelligent packaging system based on counting and classification of surgical instruments, relates to the technical field of medical equipment management, and aims to solve the technical problems of high error rate, low packaging efficiency and difficulty in instrument management traceability in the conventional manual counting process in the surgical instrument management process. The multi-modal image acquisition module is used for acquiring multi-modal image data and related auxiliary information of surgical instruments, and comprises an algorithm processing anti-interference module, a data verification and correction center, an instrument process management adaptation module, an information extraction and tracking module, an emergency control interaction module, an intelligent packaging reminding module, a data storage interaction module and a standard parameter library dynamic updating module. Through the standard parameter library dynamic updating module and the in-use instrument parameter regular correction mechanism, real-time synchronization of the standard parameter library and the actual instrument state is achieved, the problem of recognition precision attenuation caused by a traditional static parameter library is effectively solved, and the applicability and reliability of the management system are remarkably improved.

The invention provides a multi-modal cerebral arterial thrombosis medical image segmentation method, device and equipment, and the method comprises the steps: extracting the independent features of different modal medical images through combining a ViT encoder branch and a CNN encoder branch which are finely adjusted by a hybrid expert as a multi-modal image double-branch coding network; further integrating complementary information of different modes by using a mode missing adaptive fusion network, and performing global-local information interaction between CNN features and ViT features by using an encoder branch interaction network, so that specific features and cross-mode invariant features of different available modes can be decoupled under the condition of mode missing; and meanwhile, the advantages of different types of features are fully utilized, and the value information of the multi-modal image features is deeply mined, so that accurate multi-modal cerebral arterial thrombosis medical image segmentation and imaging are realized, and the method is high in reliability, good in accuracy and good in practicability.

The invention discloses a text-guided multi-dimensional multi-modal image clustering method and system, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target clustering dimension and a plurality of to-be-clustered images, and generating a description text and two answer texts for the target clustering dimension for each image; performing feature coding on each image and the corresponding description text and answer text to obtain image features, description text features and answer text features; calculating similarities between the image features and the latter two, and performing weighted fusion based on the similarities to obtain fused text features; performing cross attention calculation on the image features by taking the fused text features as query to obtain image representation focused on a target clustering dimension after text guidance, and clustering according to the image representation to obtain a clustering result; the method has the advantages that the problems of text and image content disjunction and semantic dimension conflict in image clustering can be relieved, and semantic interpretability and accuracy of clustering results are improved.

A multimodal imaging unit for depth-surface imaging of a skin region of interest includes an ultrasound imaging transceiver, an optically transparent acoustic deflector, an optical module, and an optical camera sensor. An ultrasound beam emitted towards the acoustic deflector is deflected towards the skin region of interest, and an ultrasound beam reflected from the skin region of interest is returned to the ultrasound imaging transceiver. An optical beam from the optical module is passed through the acoustic deflector to a skin area of the skin region of interest and an optical beam reflected from the skin area is returned to an optical camera sensor through the acoustic deflector. The transceiver and the acoustic deflector are surrounded by an intermediary coupling medium and are hermetically enclosed by a cover provided with an acoustically and optically transparent access port for the ultrasound beams and the optical beams.