Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

873 results about "Image description" patented technology

An image description is a textual, audio or graphical content portraying the image in a representation intelligible by the addressees. The description should be comprehensive as well as perceptible by the target audience.

Vision-text collaborative abstract generation method and system based on multi-modal learning

The invention belongs to the field of natural language processing crossing, and particularly relates to a visual-text collaborative abstract generation method and system based on multi-modal learning, and the system comprises a multi-modal data receiving module which is used for receiving multi-modal input data in parallel, and the multi-modal input data comprises text data and visual data; the visual semantic understanding module is used for extracting high-level semantic features of the image by adopting a visual semantic understanding model and generating text description; the semantic fusion module is used for carrying out semantic fusion on the original text and the generated image description by adopting a vision-text semantic fusion method based on direction consistency and self-adaptive semantic completion; and the abstract optimization module is used for carrying out semantic reconstruction on the fusion features by adopting a multilayer deep fusion hybrid neural network architecture, and generating an abstract text conforming to context semantics and accurate expression. The method has the advantages that visual and text information is accurately aligned, high-quality abstracts are generated, and the method is particularly suitable for news reports, conference records, video content analysis and other scenes needing cross-modal information fusion.
Owner:SHANDONG UNIV +1

Knowledge base question and answer method and device and computer readable storage medium

The invention relates to a knowledge base question answering method and device and a computer readable storage medium, and belongs to the technical field of artificial intelligence. A query question input by a user is received; the query problem is analyzed, and keywords in the query problem are extracted; retrieving slice data matched with the query problem from a retrieval enhancement generation system on the basis of the text correlation between the keyword and the slice data; wherein a plurality of slice data are pre-stored in the retrieval enhancement generation system, the slice data comprise text slices and image-text slices, and the image-text slices comprise storage paths of images and image descriptions with text context information of the images; and inputting the retrieved slice data into a multi-modal large model, generating an answer related to the query question and an image matched with the answer, and outputting the question answer with the image. According to the method, the capability of an original retrieval enhancement generation system is remarkably improved, and the information requirement of a user in an image-text environment is more comprehensively met.
Owner:BEIJING KNOWLEDGE ATLAS TECHNOLOGY CO LTD

Generation and evaluation method for obtaining long text description from laparoscopic surgery image

The invention discloses a generation and evaluation method for acquiring long text description from a laparoscopic surgery image, which comprises the following steps of: constructing a cue word containing the laparoscopic surgery image, short text description, a target bounding box and surgery knowledge, and constructing long text description data by utilizing a large language model based on the cue word; constructing an image long text description generation model which comprises a pre-trained visual encoder, a visual feature query converter, a language decoder and an external operation concept vector database; performing supervised training on the image long text description generation model by using the long text description data; constructing a long text description evaluation method to evaluate the trained model; the method comprises the following steps: acquiring an input laparoscope image video stream through a laparoscope image system, and screening a key frame as a to-be-processed image; and inputting a to-be-processed image into the trained and evaluated model to obtain an image long text description result. According to the method, the ability of understanding operation details in the image can be enhanced, and more comprehensive operation image description can be generated.
Owner:ZHEJIANG UNIV

Multi-modal literature data extraction method and device and medium

The invention discloses a multi-modal literature data extraction method and device and a medium. The method comprises the following steps: firstly, standardizing literatures to obtain a structured document comprising a literature directory, author information, a paragraph text, a paragraph layout, a table LaTeX code, a formula LaTeX code and / or image description; then, related retrieval of fragments is performed on texts, formulas, tables and images from the structured document according to keywords and / or vectorization retrieval strategies. The retrieved fragments are optimized through correlation sorting, and the number of the fragments needing to enter large model processing is reduced. And finally, performing question and answer configuration on the large language model based on a user extraction demand and a sorting fragment, generating a structured answer through a prompt engineering technology, ensuring that the results can be traced through an original fragment, ensuring the availability of question and answer results, and reminding the user to verify and process a part which cannot be traced when necessary.
Owner:ZHEJIANG LAB

Intelligent coagulant adding control method and system based on image recognition and multi-parameter modeling

The invention relates to an image processing and data processing technology, in particular to an intelligent coagulant dosing control method and system based on image recognition and multi-parameter modeling, the floc state is accurately quantified through image recognition and deep learning modeling, and a dosing prediction model with self-adaptive capacity is established in combination with raw water feed-forward information. And accurate control of coagulant addition is realized. The method comprises the following steps: collecting a floc image, carrying out image processing and floc feature extraction, and constructing a floc image description vector; time sequence input structure data fusing the floc image and the water quality data is constructed, and floc image sampling at each moment is defined as a time frame; performing enhancement and reconstruction processing on the training data of the dosing amount prediction model by adopting a data enhancement and sample equalization strategy to obtain continuously distributed synthetic samples; and constructing a hierarchical feature fusion enhanced dosing amount prediction model, fusing the previous water quality parameters, the current water quality parameters and the floc image joint feature vectors, and optimizing the dosing amount prediction precision layer by layer.
Owner:GUANGDONG LONGQUAN TECH CO LTD

Chatbot response system

A computer-implemented method and system for responding to user posts containing images with relevant image responses during conversation between a user and a chatbot. The system receives an image post from the user and generates a description of the image using an image-to-text model. User intent is determined based on the image and description. If responding with an image is appropriate based on the user intent, the system generates a prompt using the image description and passes it to a text generation model to create an image description and caption. The image description and caption are used to synthesize a new image. The resulting image and caption are packaged into a post that is provided as a response to the user. The system uses machine learning pipelines and models to analyze images, detect inappropriate content, classify user intent, generate text, and synthesize images.
Owner:SNAP INC

Early warning method and device based on multi-modal large model, equipment and medium

The invention relates to an early warning method and device based on a multi-modal large model, equipment and a medium, and belongs to the technical field of deep learning. The method comprises the steps of collecting a real-time video stream; receiving the real-time video stream, and performing frame extraction operation on the real-time video stream to obtain a key frame image; calling a classification model to classify the key frame images, and screening out to-be-analyzed images needing further early warning analysis; inputting the to-be-analyzed image into a multi-modal large model, understanding the content of the to-be-analyzed image to generate image description, and generating an early warning result of the to-be-analyzed image in combination with the information in the constructed knowledge base and the image description; and outputting the obtained early warning result in real time. According to the method, the classification model is adopted to perform preliminary screening on the images, the images which do not need early warning obviously are screened out in advance, and the images which need to be further screened enter the multi-mode large model to be analyzed, so that the cost of the system can be effectively reduced.
Owner:ZHEJIANG ZHIPU XINPIAN TECHNOLOGY CO LTD

Picture editing method and system based on multi-modal condition adaptation

The invention provides a picture editing method and system based on multi-modal condition adaptation, and the method comprises the steps: obtaining a first text vector: obtaining a picture description corresponding to a picture through processing, processing the picture description through an editor, and obtaining a first text vector; obtaining a first text vector based on the picture information; a second text vector acquisition step: processing the used editing instruction to obtain a second text vector based on the editing instruction; a fusion feature acquisition step: fusing the first text vector and the second text vector through weight to obtain a fusion feature; an image potential feature code acquisition step; in the image editing step, the injected condition information is received, meanwhile, the received potential features and potential noise of the image are denoised, and the image desired by the user is gradually generated in the iterative denoising process under the guidance of the received condition information; and an image restoration step. According to the invention, the stability, controllability and accuracy of image editing can be improved.
Owner:SHENZHEN EMDOOR DIGITAL TECH

Method and system for generating controllable high-quality AI drawing picture description

The invention discloses a generation method and system for controllable high-quality AI drawing picture description, and the method comprises the steps: carrying out the analysis and recognition of the drawing intention of a user through a natural language processing technology, and extracting key drawing elements from the drawing intention; the meanings of the elements and the relation between the elements are understood on the basis of a deep learning neural network model, and the drawing intention of the user is expanded and supplemented on the basis of a generative adversarial network, so that various drawing cue words in different directions are generated; and on the basis of the drawing cue word, generating an image according with the description of the drawing cue word by utilizing a diffusion model image generation algorithm, and obtaining various styles of images according with an expected quality standard by introducing an image quality evaluation system. The method has the advantages that user thinking divergence can be guided, creativity is stimulated, effective interaction is achieved, feedback optimization results are fully utilized, image quality is greatly improved, and generation diversity and flexibility are increased to adapt to various requirements and scenes.
Owner:NEW ONE (BEIJING) TECH CO LTD

Modal alignment and multi-scale extraction remote sensing image description generation method, system and device based on remote sensing image-text comparison pre-training features and medium

ActiveCN119131196ASuitable for remote sensing description generation tasksreduce utilizationDrawing from basic elementsNeural learning methodsData setWord list
The invention discloses a modal alignment and multi-scale extraction remote sensing image description generation method, system and device based on remote sensing image-text comparison pre-training features and a medium. The method comprises the steps of obtaining a remote sensing subtitle generation data set NWPU-Captions; preprocessing texts and images of a training set, a verification set and a test set of the remote sensing subtitle generation data set NWPU-Captions to obtain word lists, text features and image features; building a remote sensing subtitle generation network based on a Transform model; training the remote sensing subtitle generation network based on the Transform model by using the training set, inputting the verification set into the models obtained in each round for verification, and selecting the model with the best effect; inputting the image features in the test set into the model with the best effect to obtain subtitles corresponding to the image; the system, the equipment and the medium are used for implementing the method. Compared with a pre-trained image encoder, a multi-scale fusion extraction module and a text encoding module, features with more semantics and more scales can be extracted, and generated descriptions can contain semantic vocabularies with higher levels and more details.
Owner:XIDIAN UNIV

Short video popularity prediction method based on multi-modal retrieval enhancement

The invention belongs to the technical field of information propagation, and discloses a short video popularity prediction method based on multi-modal retrieval enhancement, which comprises the following steps: extracting key frames from a video to be predicted, and obtaining the image description of each key frame; cascading the original text information of the to-be-predicted video with the image description of the key frame of the to-be-predicted video to form a text prompt of the to-be-predicted video; inputting a text prompt of a to-be-predicted video into the large language model to obtain a retrieval vector of the to-be-predicted video; retrieving a plurality of related videos from the retrieval pool; performing feature extraction on the video to be predicted and the related video obtained by retrieval by using a pre-trained large model, and respectively extracting text and visual features; performing cross-modal feature interaction on the to-be-predicted video and the related video to obtain a corresponding mode, and performing interaction to obtain a retrieval enhancement vector; and predicting the popularity of the short video according to the modality of the video to be predicted, the modality of the related video and the retrieval enhancement vector. According to the method, retrieval enhancement is introduced, so that the prediction accuracy is greatly improved.
Owner:UNIV OF ELECTRONICS SCI & TECH OF CHINA

Intelligent hardware grinding system and method based on machine vision

The invention discloses an intelligent hardware grinding system based on machine vision, which comprises a machine vision module used for carrying out image acquisition on grinding of hardware and generating surface feature information and grinding state image feature information of the hardware; the robot grinding module is used for grinding the hardware according to the surface feature information and the grinding state image feature information; the control module is connected with the machine vision module and the robot grinding module and used for adjusting grinding equipment control instruction features and grinding parameters in real time according to the surface feature information and the grinding state image information; according to the method, the target hardware grinding information is subjected to image recognition, the image description set is mined, and accurate path planning is generated and covers shape, size, defect position and other information. According to the method, deviation judgment is carried out on the multiple grinding task events, so that the deviation correction execution strategy is determined, the hardware grinding precision is effectively improved, the hardware rejection rate is reduced, and the hardware production efficiency is improved.
Owner:JINHONGXING (HUIZHOU) TECH CO LTD

Target detection method and device, model training method and device, electronic equipment and medium

The invention relates to the technical field of data processing, and provides a target detection method and device, a model training method and device, electronic equipment and a medium. The target detection method comprises the steps that a to-be-recognized image and a query text are acquired, and the query text is used for querying a target object corresponding to the query text in the to-be-recognized image; performing image recognition on the to-be-recognized image to obtain image description features and region detection visual features; performing regional multi-modal fusion processing on the image description features and the regional detection visual features to obtain regional multi-modal fusion features; performing feature fusion processing on text features obtained based on the query text and the regional multi-modal fusion features to obtain text regional fusion features corresponding to the query text; and a target detection result is obtained based on the text features and the text region fusion features, so that the fusion degree of text semantics and image region features is improved, and the accuracy and robustness of target detection in a complex scene are improved.
Owner:BEIJING JIZHI DIGITAL TECH CO LTD

Intellectual visual question and answer method and device based on big and small model collaboration and medium

The invention discloses a knowledge-based visual question-answering method and device based on big and small model collaboration and a medium, belongs to the technical field of visual question-answering, and solves the problem of how to improve the accuracy of complex questions faced by a visual question-answering technology. In the image description extraction step, an image description and key object label set which is highly associated with a natural language problem is generated, and in the entity enhancement processing step, key entities are extracted and a clarity problem and an entity example which are used for resolving semantic ambiguity are generated. In the candidate answer generation step, a plurality of semantically complementary answers are generated under different reasoning dimensions on the basis of a multi-round generation strategy, optimized candidate answers are screened on the basis of a model built-in scoring mechanism, in the context example retrieval step, examples most related to current input are retrieved, and finally, the candidate answers are obtained. And the large language model receives a unified input prompt formed by splicing output results in the steps, and inference is performed in an autoregression mode to generate answers, so that the accuracy of the visual question-answering technology facing complex questions is effectively improved.
Owner:STATE GRID ANHUI ELECTRIC POWER CO LTD ELECTRIC POWER SCI RES INST

Multi-language object illusion relieving method based on cross-language attention mode

The invention discloses a multi-language object illusion relieving method based on a cross-language attention mode, solves the problem of how to relieve multi-language object illusion during detection of a large visual language model under non-English questions, and belongs to the technical field of multi-mode questions and answers. The method comprises the following steps: identifying a cross-modal attention head set of which a visual language model shows obviously different behaviors for English and a target language when same semantic questions in different languages are processed; constructing image description queries of English and target languages for the same image, respectively inputting the image description queries into respective visual language models for reasoning, obtaining attention output under the English and target languages, and taking an average difference between the attention output as a language migration vector of the target languages; and in the reasoning process of the target language question, intervening the attention heads in the attention head set by using the language migration vector, so that the visual understanding ability of the visual language model under the non-English question is closer to the English question.
Owner:HARBIN INST OF TECH

Image processing model

A method comprising generating image descriptions of images in an original training set of images; determining, using at least one LLM, at least one domain and / or class which is under-represented in the original training set; generating, using a second LLM and based on the determination of the at least one domain and / or class, at least one instruction for a third LLM to generate at least one text prompt; generating, using the third LLM and based on the at least one instruction, the at least one text prompt for a text-to-image model; generating, using the text-to-image model and based on the at least one text prompt, at least one synthetic image; and generating an enhanced training set of images for use in training an image processing machine learning, ML, model, the enhanced training set of images comprising the original training set of images and the at least one synthetic image.
Owner:FUJITSU LTD +1

Method and device for multi-stage generation of medical image question and answer thinking chain data

The invention discloses a method and device for generating medical image question and answer thinking chain data in multiple stages. According to the method, by introducing a multi-stage model collaborative reasoning mechanism, extraction of key image content, enhanced generation of image description and construction of a thinking chain are completed in sequence, so that the accuracy, pertinence and interpretability in a medical image question and answer task are improved; the problems that an existing model is inaccurate in image detail recognition, opaque in reasoning process, generalized in question and answer content and the like are solved.
Owner:HANGZHOU DIANZI UNIV

Multi-modal fault diagnosis method based on cross-modal data enhancement

The invention relates to a multi-modal fault diagnosis method based on cross-modal data enhancement, and the method comprises the following steps: obtaining an industrial multi-modal data set, inputting a multi-modal named entity recognition model, and obtaining a fault diagnosis result; in the multi-modal named entity recognition model, a cross-modal data enhancement module is established, and image description is generated according to image data to improve the diversity of training data, so that the robustness of the model in a low-resource scene is enhanced; constructing a global alignment optimizer based on an information theory principle, aligning global semantics among modals by maximizing consistency information among the modals, and filtering irrelevant visual noise by minimizing irrelevant information of image modals; a fine-grained cross-perception module is constructed based on a cross attention mechanism, accurate interaction between text and image modalities is captured, and multi-modal representation with rich information is generated, so that the accuracy and reliability of fault diagnosis are improved. Compared with the prior art, the robustness and accuracy of fault diagnosis can be further improved.
Owner:TONGJI UNIV

Image description generation method and system based on enhanced fine-grained information

The invention discloses an image description generation method and system based on enhanced fine-grained information. The method comprises the following steps: 1) extracting regional features and global features of an image, and carrying out weighted fusion to generate multi-view feature information; 2) combining text feature extraction and cosine similarity screening, dynamically fusing text-image features, and optimizing cross-modal features through a dynamic multi-view enhancement mechanism; (3) a hierarchical memory enhancement encoder is adopted, multi-head attention and Transform structure hierarchical coding features are combined, and detail information is reserved through a memory unit; and 4) aggregating image and text weights in two stages through a double cross-modal attention decoder to generate high-quality description. The system comprises a feature extraction module, a text processing module, an encoding module and a decoding module. Through fine-grained feature enhancement and cross-modal interaction, the problems of detail loss and semantic deviation in a traditional method are effectively solved, and the method is suitable for image semantic analysis of complex scenes.
Owner:NANJING UNIV OF SCI & TECH

Pedestrian re-identification method combined with text guidance

The invention discloses a pedestrian re-identification method combined with text guidance, which comprises the following steps: collecting diversified pedestrian images, extracting image features of the images by using a deep convolutional neural network, directly generating text description from an original image through an image description model, extracting text features, and optimizing the comparison loss of the images and the text to obtain the pedestrian re-identification result. Aligning the visual features and the text features; re-recognition classification loss is optimized, it is ensured that the model can effectively distinguish identities of different pedestrians, the performance of the model is evaluated on a verification set, fine adjustment is conducted according to feedback, and the trained model is used for pedestrian recognition. According to the method, the training of the model is supervised by introducing text description, so that the limitation of a traditional pedestrian recognition technology on capturing detail features is solved. According to the method, text description (such as garment styles, colors, textures and the like) is used for guiding and optimizing the training process of a deep learning model, and it is ensured that the model can more accurately learn and recognize the detail features.
Owner:南京行者易智能交通科技有限公司

Dynamic split mirror generation system and method based on controllable diffusion model

The invention discloses a dynamic split mirror generation system and method based on a controllable diffusion model, and belongs to the technical field of film and television production. The implementation method comprises the following steps of: 1, setting a keyword text by a user, and inputting the keyword text into ChatGPT to generate a script; 2, converting the script into a scene category, a camera motion mode, a character role position and action and image description in a shot language by utilizing ChatGPT; 3, using a CLIP model to carry out contrast training on the image encoder and the text encoder; 4, performing an OpenPose model on the action reference image to obtain skeleton key points of the character role, and converting the skeleton key points into image character actions; 5, performing action fine-grained control on the character action of the image by using a Stable Diffusion model and a ControlNet model, and generating a split image; 6, utilizing a Pika Labs model to generate a dynamic video from the split image; compared with the prior art, the accuracy of user role action matching under the scene based on multi-text and high-complexity actions is improved.
Owner:BEIJING UNIV OF POSTS & TELECOMM +2

Processing method for medical report structured information extraction and privacy protection

The invention discloses a processing method for structural information extraction and privacy protection of a medical report, which relates to the field of medical report processing and can efficiently and accurately extract text information in a table by integrating advanced image preprocessing, dynamic layout analysis, OCR (optical character recognition) technologies and a dual-channel privacy detection model. Establishing a coordinate mapping relation between the image description paragraph and the corresponding image; the generated JSON structure containing the text, table and image-text mapping relation ensures the integrity and accuracy of the data; the problems of data dislocation, context splitting, privacy disclosure and the like when the medical document with complex typesetting is processed in the prior art are solved; besides, an improved Transform framework is constructed on the basis of a Qwen model, a logical reasoning chain is injected, meanwhile, a case reading large model is obtained by integrating clinical diagnosis rules and multi-modal contrast learning strategy training, the model is utilized to generate a high-quality structured abstract, and the abstract generation accuracy is greatly improved.
Owner:SHANGHAI HENGFANG HEALTH TECHNOLOGY CO LTD

Community data management system for intelligent old-age care community

The invention relates to the field of data management, in particular to a community data management system for an intelligent old-age care community, and the system comprises a video monitoring unit which comprises a plurality of community monitoring devices and is used for collecting monitoring videos of all monitoring areas; the video analysis unit is used for periodically determining the video frame category of each basic video frame according to the personnel change reference value and the personnel influence reference value of the basic video frame corresponding to the community monitoring equipment; the video optimization unit is used for determining the state of the monitoring equipment according to the first-class video frame proportion and the first-class video frame density corresponding to the community monitoring equipment, and determining a corresponding video frame optimization mode as compensation optimization or activity feature optimization according to the state of the monitoring equipment; the key frame optimization unit is used for determining a processing mode to supplement or generate image description for the associated frame according to the effective state; and the monitoring data processing efficiency is improved.
Owner:NINGBO DAHONGYING UNIV

Image conversion method and system, computer equipment and storage medium

The invention relates to an image conversion method and system, computer equipment and a storage medium. Comprising the following steps: acquiring an image description text of an original image, and determining an original text embedding vector and a target text embedding vector according to a target description text and the image description text through a text encoder; determining a fusion text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model; and through a stable diffusion model, performing image conversion on the original image according to the fusion text embedding vector, and determining a target image. According to the scheme, the feature structure of the original image can be kept as much as possible in the image conversion process of the original image, the accuracy of the image conversion result is improved, and the obtained target image has expected visual impression.
Owner:HANGZHOU WOYING NETWORK TECH CO LTD

Image generation method and device, electronic equipment and computer readable storage medium

The embodiment of the invention discloses an image generation method and device, electronic equipment and a computer readable storage medium. According to the embodiment of the invention, after the original image and the image description text corresponding to the original image are acquired and word segmentation is performed on the image description text to obtain the text word set, feature extraction is performed on the original image, at least one negative word is generated based on the extracted image features, and feature extraction is performed on the text word set and the negative words respectively; obtaining positive text features corresponding to the text word set and negative text features corresponding to the negative words, then adjusting the positive text features by using the negative text features to obtain target text features, and generating a target image corresponding to the original image based on the target text features and the image features; according to the scheme, the image generation efficiency can be improved.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Image description method, device and equipment and computer readable storage medium

The invention discloses an image description method, device and equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: describing original image data based on a preset to-be-described feature, and obtaining an initial description text, the initial description text represents description information corresponding to features to be described in the original image data; semantic enhancement processing is conducted on the initial description text from preset five-sense dimensions, an enhanced description text is obtained, and the five-sense dimensions comprise the visual dimension, the tactile dimension, the auditory dimension, the taste dimension and the olfactory dimension; and generating comprehensive description information of the original image data based on the initial description text and the enhanced description text. According to the method and the device, the performance of the multi-modal large model in the cross-modal generation task can be improved.
Owner:ZHONGDIAN DATA IND CO LTD +1

Character image generation method and device, electronic equipment and storage medium

The embodiment of the invention provides a character image generation method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the fields of financial science and technology and medical science and technology. The method comprises the following steps: carrying out semantic feature coding on a current image description text to obtain description text semantic features; and carrying out image element identification on the semantic features of the description text to obtain image components. And updating the semantic features of the description text according to the historically generated image and the image composition elements to obtain the current semantic features. Performing image screening according to the current semantic feature to obtain a reference character image, and performing image feature coding on the reference character image to obtain a reference image feature; and performing feature fusion according to the description text semantic features and the reference image features to obtain guidance fusion features. And performing image generation based on the guidance fusion features. According to the embodiment of the invention, the accuracy of generating the character image can be improved.
Owner:PING AN TECH (SHENZHEN) CO LTD

Image editing and generating method and device, electronic equipment and storage medium

The embodiment of the invention discloses an image editing and generating method and device, electronic equipment and a storage medium, and relates to the technical field of image generation, and the method comprises the steps: obtaining a to-be-processed image and a target instruction, constructing an image description of the to-be-processed image according to a scene in the to-be-processed image, converting the image description into a text mark, and storing the text mark in the target instruction; the method comprises the steps of obtaining a to-be-processed image, embedding a text mark into a high-dimensional space through an embedding model to obtain text embedding, obtaining an element vector of the to-be-processed image according to the to-be-processed image and the text embedding through a diffusion model, designing an editing operator according to a target instruction, updating the element vector by using the editing operator, and generating a result image according to the updated element vector through the diffusion model. According to the method, efficient, accurate and flexible editing of the image is realized by combining the diffusion model and the editing operator, and the problems that the expression of the model on a specific target category cannot be improved, the problem of long tail distribution cannot be solved and data similar to a real scene cannot be generated in the prior art are solved.
Owner:SHENZHEN TIANHAI CHENGUANG TECH CO LTD

Multi-modal data generation and fine adjustment method for domain image

The invention provides a multi-modal data generation and fine tuning method for a domain image, and belongs to the technical field of artificial intelligence, and the method comprises the steps: carrying out the primary processing of the domain image through a visual multi-modal model, obtaining a general description, carrying out the natural language conversion of a set label, obtaining a conversion text, and carrying out the fine tuning of the conversion text; carrying out semantic integration and expansion on the image description and the converted text based on a large language model to generate an initial description; performing multi-layer extraction on the domain image according to description dimensions to obtain layer description of each layer, and generating comprehensive description in combination with the initial description; matching the domain image with the corresponding comprehensive description to obtain multi-modal data; and performing fine tuning processing on the visual multi-modal model by using the multi-modal data to obtain an optimized multi-modal model. And the identification and question-answering capabilities in the domain image are improved.
Owner:BEIJING ZHONGKE STRON CLOUD INTELLIGENT TECH CO LTD

Document language image-text data set construction method and device for internet data and medium

The invention relates to a method and equipment for constructing a minority language image-text data set for internet data and a medium, and the method comprises the steps: obtaining an HTML webpage file, extracting a multi-modal document containing minority language image-text data from the HTML webpage file, and screening out image-text related pairs according to replaceable texts corresponding to images in the multi-modal document; inputting a pure picture which does not contain a replaceable text in the multi-modal document, the text information after the picture is removed and a self-defined cue word into a first multi-modal large model, generating corresponding picture description information and image-text question and answer information, forming an image-text description pair by the pure picture and the picture description information, and displaying the image-text description pair in the first multi-modal large model; forming a visual question-answer pair by the pure picture and the image-text question-answer information; inputting the multi-modal document, the image-text related pair, the image-text description pair and the visual question-answer pair into a second multi-modal large model, and outputting an image scene label; and combining the information to obtain a small language image-text data set. Compared with the prior art, the method has the advantages that high-quality, diversified and standardized small language image-text data set construction can be realized.
Owner:SHANGHAI ARTIFICIAL INTELLIGENCE INNOVATION CENT