2612 results about "Scene graph" patented technology

A real-time system for the multimodal generation of virtual reality scenes based on artificial intelligence for the creation of immersive three-dimensional environments from natural language narratives, consisting of: a speech capture module configured to continuously record a user's spoken narrative via one or more directional microphones, preprocesses the captured signal by noise reduction and temporal alignment, and outputs a digital speech stream; A speech-to-text processing unit that is operationally coupled to the speech capture module and configured for real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining contextual continuity throughout the evolving narrative. a semantic interpretation processing unit that is communicatively linked to the speech recognition unit and configured to perform natural language understanding techniques to extract contextual entities, spatial references, temporal relationships, and object attributes from the transcribed narrative; the engine includes a large language model that is fine-tuned for spatial reasoning tasks; a scene graph generation module configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; a multimodal image-language model processor coupled with the scene graph generation module, wherein the processor is configured to retrieve, adapt, or synthesize appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and align these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller configured to create a cohesive virtual scene from the aligned assets, perform real-time rendering using a GPU-accelerated ray tracing pipeline, and produce a stereoscopic visual output that corresponds to the evolving narrative; A head-mounted virtual reality visualization device connected to the rendering engine and configured to display the generated immersive environment to the user in real time. The device features motion sensors and inside-out tracking cameras to detect head and body movements, dynamically updating viewing angles and perspective within the rendered scene; and a bidirectional feedback module integrated into the head-mounted device and connected to the semantic interpretation processing unit; the module is configured to interpret corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization; The system continuously updates the virtual scene as the narrative develops, ensuring temporal synchronization between speech input and rendered output below a defined latency threshold, thus enabling a natural, dialogic construction of complex three-dimensional virtual environments.

The invention discloses a community intelligent monitoring and emergency linkage method and system fusing BIM spatial semantics, and the method comprises the steps: constructing a BIM scene map, and obtaining the attributes and mutual relationships of components and spatial regions in a BIM model; mapping a dynamic target detected in video monitoring into the BIM model, and obtaining spatial semantic information of the dynamic target; based on BIM spatial semantic information of a dynamic target, a target-environment interaction graph is constructed, a graph neural network model is used for training and reasoning, and specific complex events related to spatial contexts are recognized; taking the BIM model as a space-time reference, fusing multi-source heterogeneous data, and reconstructing by adopting a graph-based event association algorithm to form a complete event chain containing an atomic event sequence and an association relationship; and when an emergency event or an event chain is detected to indicate an emergency state, combining BIM preset information and real-time sensor data, dynamically generating an optimal emergency plan, and performing visual commanding and dispatching through a BIM three-dimensional scene and augmented reality.

The invention provides a virtual reality building roaming method and system based on scene rendering, and relates to the technical field of virtual reality, and the method comprises the steps: dividing building three-dimensional model data through an octree space segmentation algorithm, and determining a detail level threshold according to a viewpoint distance, and constructing a multi-level detail model. And baking the basic illumination chartlet by using a pre-calculation irradiance transmission technology, and performing real-time synthesis in combination with the multi-resolution shadow chartlet. And calculating illumination by adopting a physics-based rendering method, and fusing the illumination with the scene light and shadow synthesis data to generate dynamic illumination rendering data. And determining a current viewpoint based on the spatial position and attitude data of the virtual reality head-mounted display, performing view cone elimination and shielding elimination, and performing GPU instantiation rendering on visible building data. And the rendered image is optimized through high dynamic range processing and a time domain anti-aliasing technology, and a final scene image is generated in combination with an atmospheric scattering algorithm. According to the invention, the rendering efficiency and scene reality sense of virtual reality building roaming can be effectively improved.

The invention provides an agent strategy generation and online optimization method based on dynamic scene perception, and the method comprises the steps: obtaining a scene demand description of a user, and carrying out the analysis of the scene demand description, so as to generate a target task sequence which can be executed by an agent disposed in a target scene; dynamically sensing the current environment characteristics of the target scene to generate a dynamic semantic topology network and generate a dynamic scene graph according to the dynamic semantic topology network; on the basis of the dynamic scene graph, hierarchical modeling of the incidence relation is carried out on the behavior space corresponding to the intelligent agent, and behavior semantic features containing scene perception are generated; the behavior semantic features containing scene perception are mapped to an intelligent agent strategy representation space, and a behavior feature strategy for controlling an intelligent agent to execute a target task sequence is obtained; and according to the determined scene value representation, decomposing the behavior feature strategy to obtain an advantage estimation value adapted to the scene so as to carry out online optimization on the behavior feature strategy. According to the invention, the intelligent agent strategy can accurately adapt to the requirements in the business process of an enterprise.

The invention discloses an image semi-supervised semantic segmentation method and system based on pixel-level correction, and relates to the technical field of computer vision and semantic segmentation, and the method comprises the steps: constructing a training data set which comprises a labeled image and an unlabeled image, and the labeled image is an image labeled with a pixel-level label; carrying out random diversity image enhancement to obtain a weakly-enhanced labeled image, a weakly-enhanced unlabeled image and a strongly-enhanced unlabeled image; constructing a teacher-student model; training a teacher-student model by using the training data set, constructing a total loss function based on a supervised loss function, a cross-image unsupervised loss function and an agent-related loss function, and optimizing and updating network parameters of the teacher-student model through the total loss function until the total loss function is minimized, so as to obtain the teacher-student model. Obtaining a trained teacher-student model; and inputting a to-be-segmented scene image into the trained student model, and obtaining and outputting a more accurate image semantic segmentation result.

The invention discloses a mechanical arm motion control method based on multi-agent cooperation, and the method comprises the steps: firstly, receiving an RGB image through a sub-task generation agent, and generating a structured sub-task sequence according to a natural language task instruction of the RGB image; secondly, performing joint modeling on a task text and a scene image through a 3D sensing intelligent body, positioning specific coordinates of a target object in a three-dimensional space, reasoning dynamic characteristics of a current environment based on historical state information of a robot by combining an environment sensor, and generating an environment sensing vector; and finally, the action generation agent performs fusion modeling according to the subtask text, the subtask target coordinates, the current state of the robot and the environment perception vector, generates a continuous action vector, drives a mechanical arm to complete each subtask action, and constructs closed-loop feedback by a controller and a discriminator to realize task execution state judgment and automatic circulation. The precise action control instruction can be effectively generated, and the task execution fineness of the mechanical arm is remarkably improved.

The invention discloses a saliency target detection method for an intelligent auxiliary driving system, and the method comprises the following steps: 1, obtaining a driving scene image based on a driver visual region in real time through a vehicle-mounted image collection device, and carrying out the preprocessing of the image; step 2, performing bottom-layer multi-scale feature extraction, interaction and global enhancement of feature information on the backbone network, and extracting rich multi-scale feature information; step 3, the neck network comprises an AESwinTransform attention module, and a cross-layer feature guide fusion module and a multi-level feature interaction fusion module in the TFG network, and the multi-level features extracted from the backbone network are fused together; step 4, detecting and classifying the feature map; and 5, inputting the adversarial sample and the normal sample into the model in a mixed manner, and performing adversarial training by using the trained model. According to the method, the detection precision of the small saliency target can be improved, the generalization ability and robustness of the model in a complex environment are improved, and the interference of irrelevant scene information is reduced.

The invention provides a self-adaptive scene analysis and target generation method and system based on deep learning, and relates to the technical field of computer vision, and the method comprises the steps: extracting the depth information of a scene image sequence, constructing three-dimensional point cloud data, calculating the distance and direction matrix between objects, and generating a scene space feature vector. Meanwhile, a semantic label graph is obtained through semantic segmentation, a semantic incidence matrix is constructed, and a scene semantic vector is calculated. Inputting the two into a variational auto-encoder to obtain a target feature vector, calculating an attention score based on the target feature vector, and generating a target description vector with a weight; and finally, target contour features are generated through a probability graph reasoning network, and a final target result is generated in combination with the regional attention graph. According to the method, the accuracy and efficiency of target generation are improved.

The invention discloses a three-dimensional Gaussian sputtering method for sparse visual angle semantic priori. The method comprises the following steps of: 1, acquiring a target scene image, constructing a real image set as a data set, and manually selecting an interested object in a target scene to perform semantic three-dimensional reconstruction to obtain an initial semantic image of a sparse view angle; a multi-view image is collected, camera external parameters and scene sparse point clouds are obtained, a plurality of views are selected and input into the SAM2 segmentation model, and a view set Is with semantic images is obtained; 2, using a pre-trained SAM2 segmentation model as an interactive image sequence segmentation model, and initializing a three-dimensional Gaussian primitive according to the scene sparse point cloud; and step 3, training parameters of semantic three-dimensional Gaussian sputtering based on the trained three-dimensional Gaussian sputtering model and the interactive image sequence segmentation model, and reconstructing a target scene. According to the method, the high-quality semantic model is efficiently reconstructed. And the reconstruction result can be easily corrected through the interactive graphical interface in the training process.

The invention provides a scene adaptive adjustment method and system based on intelligent Internet of Things and meta-universe virtual-real fusion, and relates to the technical field of artificial intelligence and Internet of Things, and the method comprises the steps: collecting a scene image and environment parameter data, carrying out the decomposition and partitioning of the image, and recognizing the type of the scene, environment parameter tensors are constructed to form a digital twinborn model, the digital twinborn model is mapped to a virtual space, a proper feature vector is selected according to an identification result to generate a target feature vector, a projection position of a virtual object in an entity space is obtained, and the feature vector is optimized and then combined with the projection position to generate an enhanced feature and a mapping matrix; and generating rendering parameters through the parameter mapping network, rendering the virtual object, and displaying the virtual object on the entity space interface in an overlapping manner.

The invention discloses a multi-modal thinking chain-based multi-modal large language model attribute prediction method, which comprises the following steps of: obtaining a multi-modal large language model, and forming a multi-modal attribute prediction framework together with a mask generator and a scene graph analyzer; designing an inference method of a hierarchical thinking chain, inputting an attribute prediction task into a multi-modal attribute prediction framework, extracting the hierarchy of an attribute table, and generating a hierarchical and formatted inference prompt; decomposing an attribute prediction task into hierarchical sub-tasks, and selecting a prompt method for the next step of prediction of the model; supervising a model prediction process through critical thinking; identifying object categories in the image by adopting a logic check prompt method, and extracting corresponding attribute values in combination with common knowledge in the model; and combining the generated scene graph with common knowledge in the model to check the predicted attributes. According to the method, the context understanding capability, the logic consistency, the task universality and the model reasoning efficiency of the attribute prediction task can be effectively improved.

The invention provides a natural language control method for a humanoid robot. By constructing the simple skill library and the complex skill library of the robot, the dependence on a large amount of operation data in a specific field is effectively reduced. According to the method, a large language model is used as a base of a task planner, and the capabilities of scene learning, semantic understanding and common sense reasoning of the task planner are brought into full play. By introducing a visual model in the field of computer vision, high-level semantic information and low-level spatial information of the environment can be obtained, and an accurate scene graph is constructed to serve as environment state representation. According to the voice interaction interface constructed by the invention, the naturalness of man-machine interaction is improved, so that a user can more conveniently communicate with the robot and input instructions, the user experience is effectively improved, and popularization of the humanoid robot in an actual application scene is facilitated.

The invention discloses a human abnormal behavior monitoring method based on a large-model multi-agent, which is executed by a modular multi-agent system deployed on a back-end server, obtains information through a monitoring camera, and comprises the following steps: obtaining a video stream from the monitoring camera by a sensing agent and extracting human body posture features; analyzing the key frame by a scene understanding agent by using a visual large model, and constructing a time sequence dynamic scene graph; the core reasoning agent evaluates the scene semantic conformity based on the pre-trained large model and performs abnormal preliminary judgment; performing fine-grained classification, interpretation generation and risk assessment on the abnormal behaviors; and the report and action agent generates an alarm and records event data. According to the invention, through multi-agent cooperative work and a large model technology, efficient and accurate monitoring of human abnormal behaviors is realized, and the intelligent level of the monitoring system and the abnormal behavior identification accuracy are improved.

The invention provides a video plot generation and scene synthesis method and system based on natural language processing, and relates to the technical field of video generation, and the method comprises the steps: receiving a script text to construct a multilayer scene map, extracting keywords to calculate semantic relevancy, executing feature decomposition and reconstruction to obtain a scene synthesis vector, and generating a video scene based on distance measurement. And extracting spatio-temporal features by using the feature pyramid, executing adaptive feature fusion, segmenting the video by applying a self-attention mechanism, and after a transition effect is inserted, performing style migration to output a finished video.

The invention discloses an industrial visual detection system based on multi-modal fusion, which belongs to the technical field of industrial data acquisition, processing and processing, and comprises a multi-modal data acquisition and calibration module, a scene image capture module, a visual information synthesis module, an algorithm optimization and training module and a multi-modal image output module, the multiple modules are integrated into the industrial visual detection system for multi-modal data acquisition, processing and fusion, high-quality original data are provided for subsequent image analysis, images of industrial scenes can be automatically captured under different illumination conditions, an original image database containing various illumination and scene conditions is constructed, and the visual detection system has the advantages of being simple in structure, convenient to operate and high in practicability. The system can operate stably, high-precision analysis and processing of the image data set in the industrial scene are realized, the corresponding fusion image is generated, the industrial visual inspection report is generated based on the fusion image, and the adaptability and reliability of the industrial visual inspection system are improved.

The invention relates to a multi-mode large language model fine tuning method, system and device and a medium, and belongs to the technical field of artificial intelligence and computer vision crossing. The fine tuning method comprises the steps that an original business scene image is acquired and preprocessed, and a preprocessed image is obtained; performing bounding box coordinate labeling and semantic label definition on the entity target in the preprocessed image through a labeling tool, and outputting a structured labeling file; based on the preprocessed image and the structured annotation file, constructing a training sample set comprising multiple rounds of image-text dialogues; loading the pre-trained multi-modal large language model, configuring low-rank matrix decomposition parameters, and generating a fine tuning instruction set; and inputting the training sample set into a pre-trained multi-modal large language model, carrying out joint training operation based on the fine tuning instruction set, and outputting the fine-tuned multi-modal large language model. According to the method, the identification accuracy, the interaction capability and the system availability of the visual question-answering system in an actual application scene are improved.

The invention belongs to the technical field of automatic driving testing, and particularly relates to an accident scene generation method based on a scene knowledge graph and considering accident causes. Comprising the following steps: step 1, modeling a scene knowledge graph; 2, modeling the scene graph time sequence prediction model; 3, modeling the time sequence causal inference model; step 4, modeling the scene graph time sequence decision generation model so as to generate an accident scene; according to the method, the accident scene database with high authenticity, high diversity and accident cause consistency can be constructed under the condition that the accident scene sample data size is limited, the number of accident scenes of the same type in the automatic driving algorithm closed-loop self-evolution cloud database is efficiently expanded, the constructed scene database is used for training the automatic driving algorithm, and the accuracy of the automatic driving algorithm is improved. The adaptive capacity of the self-driving automobile to scenes with the same type of accidents can be effectively enhanced, and safe and reliable operation of the self-driving automobile in the real world is guaranteed.

The invention provides a multi-modal target detection method and device and a multi-modal recognition system. The method comprises the steps that a scene image is acquired, and the scene image is an image of a current environment acquired by adopting image acquisition equipment; acquiring a point cloud image, the point cloud image being an image of point cloud data of the current environment acquired by using a laser radar; according to the scene image and the point cloud image, fusion is carried out to obtain a target image, and the fusion mode at least comprises image fusion and / or feature fusion; and performing target identification on the target image to obtain a target object in the target image. According to the scheme, the problem of low accuracy of target detection based on single data in the prior art is solved.

The invention relates to a lightweight multi-target instance segmentation method and system for an inspection robot, and belongs to the technical field of computer vision and deep learning. The system is used for executing the method and comprises the steps of constructing training data based on a public data set; an improved lightweight YOLOv11 network is constructed, the weight is adaptively adjusted at least through a dynamic convolution module according to the input features, and collaborative optimization is carried out on feature fusion through cascade grouping attention and channel-space attention; the method comprises the following steps: training an improved lightweight YOLOv11 network to obtain an optimal weight; and deploying the trained lightweight instance segmentation model to an inspection robot, processing campus scene image data in real time, and outputting target pixel-level contour and category information. According to the method, the model parameter quantity and the calculation cost are remarkably reduced, and meanwhile, high precision and real-time performance of multi-target instance segmentation in a campus scene are achieved.

A method for executing robotic grasps includes acquiring an image of a scene with one or more objects and processing the acquired image using a trained grasping neural network. Depth and color frames of the image are passed separately through convolutional blocks of an encoder for extracting feature maps via down sampling of the respective image frames. The feature maps extracted from the depth and color frames are fused to produce a fused feature map. The fused feature map is spatially divided into patches and fed as input to a vision transformer to encode the patches based on information from other patches in the fused feature map. The output of the vision transformer is fed to a decoder to construct a grasp map of the scene. An optimal grasping location is estimated from the grasp map.

The invention relates to the technical field of computers, in particular to a multi-mode-based three-dimensional model editing method and device, equipment and a medium. The method comprises the following steps: acquiring a multi-modal fusion image personalized generation method and a source three-dimensional model; obtaining a personalized image generation network corresponding to the source three-dimensional model, and training the personalized image generation network; based on the reference image list and the corresponding text description, multi-modal embedded information is obtained after fusion; taking the trained personalized image generation network as a skeleton, and adding a multi-personalized learning module to obtain a multi-personalized image editing network; and editing the source three-dimensional scene image according to the multi-modal embedded information by using a multi-personalized image editing network to obtain a single-view-angle edited image. The multi-mode embedded information generated by the method allows a user to input prompt information of multiple modes such as texts, images, sketches and the like, complementarity of prompt of each mode is mined, so that controllability of editing is improved, and high-quality three-dimensional model editing is realized.

The invention discloses a VLM model intelligent decision-making-based driving method and device and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: extracting key visual features from continuous multi-frame driving scene images based on a preset visual feature extraction algorithm; inputting a user instruction and a time sequence visual Token corresponding to the key visual features into a preset VLM model for multi-modal alignment, and generating a target planning Token; inputting the target planning Token and the time sequence vision Token into a preset trajectory generation model to obtain a predicted trajectory; and converting the predicted trajectory into a control signal, and controlling the mobile device to complete a moving action based on the control signal. The problem of modal difference between a semantic space and an action space is solved.

The invention relates to the technical field of equipment cooperative control, and discloses a multi-equipment cooperative control method and system under an autonomous cooperation algorithm, and the method comprises the steps: collecting the physical attribute data, real-time operation data, production task data and a smart factory scene graph of operation equipment under a multi-equipment cooperative task in a smart factory application; based on the physical attribute data and the real-time operation data, identifying a space conflict mode and a time conflict mode of the operation equipment to obtain collaborative conflict information; according to the production task data and the collaboration conflict information, carrying out collaboration conflict decoupling on the operation equipment to obtain a feasible collaboration mode; performing obstacle avoidance reinforcement learning on the operation equipment by using the smart factory scene graph to construct an obstacle avoidance global path of the operation equipment in the smart factory; and based on the feasible coordination mode and the obstacle avoidance global path, constructing a target coordination control scheme of the operation equipment so as to perform coordination control on the operation equipment. According to the invention, the reliability of multi-device cooperative control under the autonomous cooperation algorithm can be improved.

The invention provides a multi-view-oriented three-dimensional scene image reconstruction registration and optimization method and system, and relates to the technical field of image processing, and the method comprises the steps: obtaining multi-frame three-dimensional scene image data, extracting a multi-level feature set, constructing a cross-view-angle semantic association graph, building a feature corresponding relation, and calculating a multi-view-angle spatial transformation relation parameter. Performing coordinate system alignment on the image data to generate an initial three-dimensional reconstruction result, and performing optimization in combination with a multi-target joint optimization function and a dynamic adaptive weight regulation and control mechanism. According to the method, the precision and robustness of three-dimensional scene reconstruction are improved, and the problem of registration errors caused by large view angle difference in a complex scene is solved.

The invention relates to the technical field of robot navigation, and discloses a laser vision fused map dynamic construction method and system, and the system comprises a data perception module, a feature analysis module, an information fusion module and a dynamic optimization module. The data sensing module collects an environment point cloud sequence and a scene image sequence through laser ranging equipment and a visual imaging device; the feature analysis module extracts spatial structure features of the point cloud and visual semantic features of the image; the information fusion module adopts a weight distribution strategy to fuse the multi-modal features to generate an initial map; and the dynamic optimization module adjusts the map position and attribute parameters in real time by using the incremental data. Through multi-modal fusion and dynamic optimization, the problems of insufficient precision, semantic deficiency and low dynamic updating efficiency of a traditional single sensor map are solved, and the method is suitable for scenes of robot navigation, automatic driving and the like.

The invention discloses an artificial intelligence-based intelligent environment simulation method and system, and relates to the technical field of artificial intelligence environment simulation, and the method comprises the steps: forming a scene tensor based on an updated initial scene and a POMDP belief state set, using an optimized VAE model to generate an initial image sequence, using a U-Net diffusion model to optimize the initial image sequence, and using a U-Net diffusion model to optimize the initial image sequence; the method comprises the steps of calculating reconstruction loss of a VAE encoder and diffusion denoising loss of a U-Net diffusion model, calculating a joint loss function of the VAE encoder and the U-Net diffusion model, generating a high-quality image sequence by using the joint loss function through a Monte Carlo simulation method, and optimizing the high-quality image sequence through a conditional diffusion model to generate a final scene image sequence. According to the method, efficient fusion of multi-modal data is achieved, the defect that in the prior art, vision and semantics are inconsistent is overcome, task specific parameters are rapidly generated through inner circulation, meta-model generalization ability is optimized through outer circulation, a model can be rapidly matched with an edge scene, and the generation efficiency and fidelity of a rare scene are remarkably improved.

The invention discloses a method for determining a preset position of a video monitoring camera. Camera presetting positions usually need to be manually set one by one, and time and labor are wasted. The method is used for a video monitoring system, and automatic setting of the preset position of the camera is realized by utilizing a group of scene images and the to-be-monitored target image, so that the efficiency is greatly improved. The method comprises the following steps: acquiring feature points based on a scene image set by adopting a computer vision technology, generating a three-dimensional model of a scene by adopting a motion recovery structure technology and a surface reconstruction algorithm, and obtaining the position of a to-be-monitored target in a three-dimensional space by matching the feature points of a to-be-monitored target image with the feature points of the scene image set. Therefore, a proper camera is selected, a corresponding preset position is calculated, and finally the preset position is corrected according to an actual monitoring effect.

The invention discloses a new view angle synthesis method and system based on a generative adversarial strategy and Gaussian sputtering. The method comprises the following steps: acquiring a scene image under a reconstructed scene view angle; based on a pre-constructed Gaussian sputtering model, obtaining an iterated intermediate rendering image according to the scene image; constructing a discriminator of a generative adversarial model, taking the Gaussian sputtering model as a generator of the generative adversarial model, and training the Gaussian sputtering model according to the scene image and the intermediate rendering image based on the discriminator and the generator to obtain an optimized Gaussian sputtering model; and obtaining a target camera pose, and inputting the target camera pose to the optimized Gaussian sputtering model to obtain a target scene image. Visual artifacts in the rendering result can be reduced, detail parts in the scene can be better fitted, and the rendering effect of scene details is enhanced.

Disclosed in the present invention are an edge-deployed semi-supervised anomaly detection method and system for a railway track foreign object. The method comprises the following steps: an edge device encoding and decoding a video stream captured by a camera to obtain an image frame sequence, and performing frame extraction; and using a semantic segmentation model to perform image segmentation on a certain image frame obtained by means of frame extraction, to obtain a railway track region segmentation image. The use of a single image as input may generate an expert model result having a high weight value; however, the determination based on a single image is not stable, multiple consecutive images of the task scene need to be inputted, the frequency of each expert model obtaining the highest weight is computed, and the expert model corresponding to the highest frequency is the final solution. The present invention supports scene-adaptive foreign object detection algorithm automatic selection, and a user can perform selection on the basis of prior knowledge, or selection may be performed by a scene-adaptive automatic algorithm selection method; the user only needs to provide a batch of image data of the current scene, and the optimal algorithm selection can be evaluated.

The invention discloses a cross-modal alignment method and device for a multi-modal large language model, and relates to the technical field of multi-modal alignment, and the method comprises the steps: obtaining an image, carrying out the target detection, and generating a local image fine-grained feature; obtaining a text, extracting text features, extracting a text entity relationship, and obtaining an entity relationship dependency tree as a text entity relationship graph; performing local fine-grained semantic alignment on the local image fine-grained features and the text features; using the scene graph model to obtain a visual relation graph between detection targets in the image; performing graph structure alignment and graph semantic alignment on the text entity relation graph and the visual relation graph; and fusing the semantic alignment features and the graph alignment features to obtain semantic alignment features with rich fine grit, global semantic alignment features and richer relationship alignment features, and inputting the semantic alignment features, the global semantic alignment features and the richer relationship alignment features into a multi-modal large model. According to the method, alignment of different modal data on different feature dimensions is realized, so that the different modal data are accurately associated.