673 results about "Image description" patented technology

The invention belongs to the field of natural language processing crossing, and particularly relates to a visual-text collaborative abstract generation method and system based on multi-modal learning, and the system comprises a multi-modal data receiving module which is used for receiving multi-modal input data in parallel, and the multi-modal input data comprises text data and visual data; the visual semantic understanding module is used for extracting high-level semantic features of the image by adopting a visual semantic understanding model and generating text description; the semantic fusion module is used for carrying out semantic fusion on the original text and the generated image description by adopting a vision-text semantic fusion method based on direction consistency and self-adaptive semantic completion; and the abstract optimization module is used for carrying out semantic reconstruction on the fusion features by adopting a multilayer deep fusion hybrid neural network architecture, and generating an abstract text conforming to context semantics and accurate expression. The method has the advantages that visual and text information is accurately aligned, high-quality abstracts are generated, and the method is particularly suitable for news reports, conference records, video content analysis and other scenes needing cross-modal information fusion.

The invention discloses a generation and evaluation method for acquiring long text description from a laparoscopic surgery image, which comprises the following steps of: constructing a cue word containing the laparoscopic surgery image, short text description, a target bounding box and surgery knowledge, and constructing long text description data by utilizing a large language model based on the cue word; constructing an image long text description generation model which comprises a pre-trained visual encoder, a visual feature query converter, a language decoder and an external operation concept vector database; performing supervised training on the image long text description generation model by using the long text description data; constructing a long text description evaluation method to evaluate the trained model; the method comprises the following steps: acquiring an input laparoscope image video stream through a laparoscope image system, and screening a key frame as a to-be-processed image; and inputting a to-be-processed image into the trained and evaluated model to obtain an image long text description result. According to the method, the ability of understanding operation details in the image can be enhanced, and more comprehensive operation image description can be generated.

The invention relates to an image processing and data processing technology, in particular to an intelligent coagulant dosing control method and system based on image recognition and multi-parameter modeling, the floc state is accurately quantified through image recognition and deep learning modeling, and a dosing prediction model with self-adaptive capacity is established in combination with raw water feed-forward information. And accurate control of coagulant addition is realized. The method comprises the following steps: collecting a floc image, carrying out image processing and floc feature extraction, and constructing a floc image description vector; time sequence input structure data fusing the floc image and the water quality data is constructed, and floc image sampling at each moment is defined as a time frame; performing enhancement and reconstruction processing on the training data of the dosing amount prediction model by adopting a data enhancement and sample equalization strategy to obtain continuously distributed synthetic samples; and constructing a hierarchical feature fusion enhanced dosing amount prediction model, fusing the previous water quality parameters, the current water quality parameters and the floc image joint feature vectors, and optimizing the dosing amount prediction precision layer by layer.

A computer-implemented method and system for responding to user posts containing images with relevant image responses during conversation between a user and a chatbot. The system receives an image post from the user and generates a description of the image using an image-to-text model. User intent is determined based on the image and description. If responding with an image is appropriate based on the user intent, the system generates a prompt using the image description and passes it to a text generation model to create an image description and caption. The image description and caption are used to synthesize a new image. The resulting image and caption are packaged into a post that is provided as a response to the user. The system uses machine learning pipelines and models to analyze images, detect inappropriate content, classify user intent, generate text, and synthesize images.

The invention provides a picture editing method and system based on multi-modal condition adaptation, and the method comprises the steps: obtaining a first text vector: obtaining a picture description corresponding to a picture through processing, processing the picture description through an editor, and obtaining a first text vector; obtaining a first text vector based on the picture information; a second text vector acquisition step: processing the used editing instruction to obtain a second text vector based on the editing instruction; a fusion feature acquisition step: fusing the first text vector and the second text vector through weight to obtain a fusion feature; an image potential feature code acquisition step; in the image editing step, the injected condition information is received, meanwhile, the received potential features and potential noise of the image are denoised, and the image desired by the user is gradually generated in the iterative denoising process under the guidance of the received condition information; and an image restoration step. According to the invention, the stability, controllability and accuracy of image editing can be improved.

The invention discloses an intelligent hardware grinding system based on machine vision, which comprises a machine vision module used for carrying out image acquisition on grinding of hardware and generating surface feature information and grinding state image feature information of the hardware; the robot grinding module is used for grinding the hardware according to the surface feature information and the grinding state image feature information; the control module is connected with the machine vision module and the robot grinding module and used for adjusting grinding equipment control instruction features and grinding parameters in real time according to the surface feature information and the grinding state image information; according to the method, the target hardware grinding information is subjected to image recognition, the image description set is mined, and accurate path planning is generated and covers shape, size, defect position and other information. According to the method, deviation judgment is carried out on the multiple grinding task events, so that the deviation correction execution strategy is determined, the hardware grinding precision is effectively improved, the hardware rejection rate is reduced, and the hardware production efficiency is improved.

The invention relates to the technical field of data processing, and provides a target detection method and device, a model training method and device, electronic equipment and a medium. The target detection method comprises the steps that a to-be-recognized image and a query text are acquired, and the query text is used for querying a target object corresponding to the query text in the to-be-recognized image; performing image recognition on the to-be-recognized image to obtain image description features and region detection visual features; performing regional multi-modal fusion processing on the image description features and the regional detection visual features to obtain regional multi-modal fusion features; performing feature fusion processing on text features obtained based on the query text and the regional multi-modal fusion features to obtain text regional fusion features corresponding to the query text; and a target detection result is obtained based on the text features and the text region fusion features, so that the fusion degree of text semantics and image region features is improved, and the accuracy and robustness of target detection in a complex scene are improved.

The invention discloses a knowledge-based visual question-answering method and device based on big and small model collaboration and a medium, belongs to the technical field of visual question-answering, and solves the problem of how to improve the accuracy of complex questions faced by a visual question-answering technology. In the image description extraction step, an image description and key object label set which is highly associated with a natural language problem is generated, and in the entity enhancement processing step, key entities are extracted and a clarity problem and an entity example which are used for resolving semantic ambiguity are generated. In the candidate answer generation step, a plurality of semantically complementary answers are generated under different reasoning dimensions on the basis of a multi-round generation strategy, optimized candidate answers are screened on the basis of a model built-in scoring mechanism, in the context example retrieval step, examples most related to current input are retrieved, and finally, the candidate answers are obtained. And the large language model receives a unified input prompt formed by splicing output results in the steps, and inference is performed in an autoregression mode to generate answers, so that the accuracy of the visual question-answering technology facing complex questions is effectively improved.

The invention discloses a multi-language object illusion relieving method based on a cross-language attention mode, solves the problem of how to relieve multi-language object illusion during detection of a large visual language model under non-English questions, and belongs to the technical field of multi-mode questions and answers. The method comprises the following steps: identifying a cross-modal attention head set of which a visual language model shows obviously different behaviors for English and a target language when same semantic questions in different languages are processed; constructing image description queries of English and target languages for the same image, respectively inputting the image description queries into respective visual language models for reasoning, obtaining attention output under the English and target languages, and taking an average difference between the attention output as a language migration vector of the target languages; and in the reasoning process of the target language question, intervening the attention heads in the attention head set by using the language migration vector, so that the visual understanding ability of the visual language model under the non-English question is closer to the English question.

A method comprising generating image descriptions of images in an original training set of images; determining, using at least one LLM, at least one domain and / or class which is under-represented in the original training set; generating, using a second LLM and based on the determination of the at least one domain and / or class, at least one instruction for a third LLM to generate at least one text prompt; generating, using the third LLM and based on the at least one instruction, the at least one text prompt for a text-to-image model; generating, using the text-to-image model and based on the at least one text prompt, at least one synthetic image; and generating an enhanced training set of images for use in training an image processing machine learning, ML, model, the enhanced training set of images comprising the original training set of images and the at least one synthetic image.

The invention discloses a method and device for generating medical image question and answer thinking chain data in multiple stages. According to the method, by introducing a multi-stage model collaborative reasoning mechanism, extraction of key image content, enhanced generation of image description and construction of a thinking chain are completed in sequence, so that the accuracy, pertinence and interpretability in a medical image question and answer task are improved; the problems that an existing model is inaccurate in image detail recognition, opaque in reasoning process, generalized in question and answer content and the like are solved.

The invention relates to a multi-modal fault diagnosis method based on cross-modal data enhancement, and the method comprises the following steps: obtaining an industrial multi-modal data set, inputting a multi-modal named entity recognition model, and obtaining a fault diagnosis result; in the multi-modal named entity recognition model, a cross-modal data enhancement module is established, and image description is generated according to image data to improve the diversity of training data, so that the robustness of the model in a low-resource scene is enhanced; constructing a global alignment optimizer based on an information theory principle, aligning global semantics among modals by maximizing consistency information among the modals, and filtering irrelevant visual noise by minimizing irrelevant information of image modals; a fine-grained cross-perception module is constructed based on a cross attention mechanism, accurate interaction between text and image modalities is captured, and multi-modal representation with rich information is generated, so that the accuracy and reliability of fault diagnosis are improved. Compared with the prior art, the robustness and accuracy of fault diagnosis can be further improved.

The invention discloses an image description generation method and system based on enhanced fine-grained information. The method comprises the following steps: 1) extracting regional features and global features of an image, and carrying out weighted fusion to generate multi-view feature information; 2) combining text feature extraction and cosine similarity screening, dynamically fusing text-image features, and optimizing cross-modal features through a dynamic multi-view enhancement mechanism; (3) a hierarchical memory enhancement encoder is adopted, multi-head attention and Transform structure hierarchical coding features are combined, and detail information is reserved through a memory unit; and 4) aggregating image and text weights in two stages through a double cross-modal attention decoder to generate high-quality description. The system comprises a feature extraction module, a text processing module, an encoding module and a decoding module. Through fine-grained feature enhancement and cross-modal interaction, the problems of detail loss and semantic deviation in a traditional method are effectively solved, and the method is suitable for image semantic analysis of complex scenes.

The invention discloses a dynamic split mirror generation system and method based on a controllable diffusion model, and belongs to the technical field of film and television production. The implementation method comprises the following steps of: 1, setting a keyword text by a user, and inputting the keyword text into ChatGPT to generate a script; 2, converting the script into a scene category, a camera motion mode, a character role position and action and image description in a shot language by utilizing ChatGPT; 3, using a CLIP model to carry out contrast training on the image encoder and the text encoder; 4, performing an OpenPose model on the action reference image to obtain skeleton key points of the character role, and converting the skeleton key points into image character actions; 5, performing action fine-grained control on the character action of the image by using a Stable Diffusion model and a ControlNet model, and generating a split image; 6, utilizing a Pika Labs model to generate a dynamic video from the split image; compared with the prior art, the accuracy of user role action matching under the scene based on multi-text and high-complexity actions is improved.

The invention discloses a processing method for structural information extraction and privacy protection of a medical report, which relates to the field of medical report processing and can efficiently and accurately extract text information in a table by integrating advanced image preprocessing, dynamic layout analysis, OCR (optical character recognition) technologies and a dual-channel privacy detection model. Establishing a coordinate mapping relation between the image description paragraph and the corresponding image; the generated JSON structure containing the text, table and image-text mapping relation ensures the integrity and accuracy of the data; the problems of data dislocation, context splitting, privacy disclosure and the like when the medical document with complex typesetting is processed in the prior art are solved; besides, an improved Transform framework is constructed on the basis of a Qwen model, a logical reasoning chain is injected, meanwhile, a case reading large model is obtained by integrating clinical diagnosis rules and multi-modal contrast learning strategy training, the model is utilized to generate a high-quality structured abstract, and the abstract generation accuracy is greatly improved.

The invention relates to the field of data management, in particular to a community data management system for an intelligent old-age care community, and the system comprises a video monitoring unit which comprises a plurality of community monitoring devices and is used for collecting monitoring videos of all monitoring areas; the video analysis unit is used for periodically determining the video frame category of each basic video frame according to the personnel change reference value and the personnel influence reference value of the basic video frame corresponding to the community monitoring equipment; the video optimization unit is used for determining the state of the monitoring equipment according to the first-class video frame proportion and the first-class video frame density corresponding to the community monitoring equipment, and determining a corresponding video frame optimization mode as compensation optimization or activity feature optimization according to the state of the monitoring equipment; the key frame optimization unit is used for determining a processing mode to supplement or generate image description for the associated frame according to the effective state; and the monitoring data processing efficiency is improved.

The invention discloses an image description method, device and equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: describing original image data based on a preset to-be-described feature, and obtaining an initial description text, the initial description text represents description information corresponding to features to be described in the original image data; semantic enhancement processing is conducted on the initial description text from preset five-sense dimensions, an enhanced description text is obtained, and the five-sense dimensions comprise the visual dimension, the tactile dimension, the auditory dimension, the taste dimension and the olfactory dimension; and generating comprehensive description information of the original image data based on the initial description text and the enhanced description text. According to the method and the device, the performance of the multi-modal large model in the cross-modal generation task can be improved.

The embodiment of the invention provides a character image generation method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the fields of financial science and technology and medical science and technology. The method comprises the following steps: carrying out semantic feature coding on a current image description text to obtain description text semantic features; and carrying out image element identification on the semantic features of the description text to obtain image components. And updating the semantic features of the description text according to the historically generated image and the image composition elements to obtain the current semantic features. Performing image screening according to the current semantic feature to obtain a reference character image, and performing image feature coding on the reference character image to obtain a reference image feature; and performing feature fusion according to the description text semantic features and the reference image features to obtain guidance fusion features. And performing image generation based on the guidance fusion features. According to the embodiment of the invention, the accuracy of generating the character image can be improved.

The invention provides a multi-modal data generation and fine tuning method for a domain image, and belongs to the technical field of artificial intelligence, and the method comprises the steps: carrying out the primary processing of the domain image through a visual multi-modal model, obtaining a general description, carrying out the natural language conversion of a set label, obtaining a conversion text, and carrying out the fine tuning of the conversion text; carrying out semantic integration and expansion on the image description and the converted text based on a large language model to generate an initial description; performing multi-layer extraction on the domain image according to description dimensions to obtain layer description of each layer, and generating comprehensive description in combination with the initial description; matching the domain image with the corresponding comprehensive description to obtain multi-modal data; and performing fine tuning processing on the visual multi-modal model by using the multi-modal data to obtain an optimized multi-modal model. And the identification and question-answering capabilities in the domain image are improved.

The invention relates to a method and equipment for constructing a minority language image-text data set for internet data and a medium, and the method comprises the steps: obtaining an HTML webpage file, extracting a multi-modal document containing minority language image-text data from the HTML webpage file, and screening out image-text related pairs according to replaceable texts corresponding to images in the multi-modal document; inputting a pure picture which does not contain a replaceable text in the multi-modal document, the text information after the picture is removed and a self-defined cue word into a first multi-modal large model, generating corresponding picture description information and image-text question and answer information, forming an image-text description pair by the pure picture and the picture description information, and displaying the image-text description pair in the first multi-modal large model; forming a visual question-answer pair by the pure picture and the image-text question-answer information; inputting the multi-modal document, the image-text related pair, the image-text description pair and the visual question-answer pair into a second multi-modal large model, and outputting an image scene label; and combining the information to obtain a small language image-text data set. Compared with the prior art, the method has the advantages that high-quality, diversified and standardized small language image-text data set construction can be realized.

The invention discloses a multi-modal large model object illusion relieving method for removing training prejudice based on efficient machine forgetting. The method comprises the following steps: S1, collecting a deviation reasoning result: firstly, selecting an instruction related to an image description task from LVLM training data as a source data set; thirdly, the image and text instructions in the source data set are subjected to reasoning again through LVLM, and a reasoning result set is generated; by comparing the object existence conditions in the real output and the reasoning result, screening out the reasoning result containing the illusion phenomenon; and S2, efficient depolarization training: based on the collected deviation reasoning result, carrying out depolarization training on the LM head by adopting a machine forgetting method based on gradient rise. The invention provides an efficient depolarization method by specifically eliminating the illusion phenomenon in a large visual language model (LVLM). According to the method, a deviation reasoning result of a model on training data is collected, and a language model head (LM head) is finely adjusted, so that efficient utilization of data and parameters is realized.

Embodiments of the present disclosure provide a scoring model training method and apparatus, an image-text pair scoring method and apparatus, a device, and a medium. The method comprises: obtaining a sample image-text pair comprising a first image description of a first language, a sample image corresponding to the first image description, and a second image description of a second language, and label data of the sample image-text pair; using the first image description to obtain a first text representation by means of a first language encoder; using the second image description to obtain a second text representation by means of a second language encoder; using the sample image to obtain a sample image representation by means of an image encoder; using the first text representation, the second text representation, and the sample image representation to obtain a first multi-modal representation by means of a multi-modal fusion network; using the first multi-modal representation to obtain a prediction score by means of a scoring network; and using a difference between the prediction score and the label data to adjust a scoring model comprising the structure, so as to perform training to obtain a scoring model capable of performing quality evaluation on an image-text pair.

The present disclosure provides a booklist cover generation method, apparatus, device, and storage medium, wherein the method includes: acquiring booklist information about a target booklist, wherein the target booklist indicates a book collection, and the booklist information represents introduction information for the target booklist; determining cover image description information for the target booklist that includes description information corresponding to at least one characteristic dimension of booklist attributes, based on the booklist information; generating a cover image for the target booklist based on the cover image description information.

The embodiment of the invention discloses a UV finish paint performance detection and analysis method and device based on image data processing, and belongs to the technical field of image analysis. According to the method, a plurality of first UV finish paint performance detection images in upstream and downstream set detection periods are obtained, and finish paint state information in different stages is comprehensively covered. And on the basis of a specific period and an image description channel mining performance representation vector set, the performances in multiple aspects such as hardness, wear resistance, glossiness and adhesive force can be accurately quantified. In-depth analysis is carried out with a small set performance analysis task period, and details of performance changes can be captured. A target performance evaluation decision algorithm debugged by a large number of training images is adopted to carry out performance evaluation decision, and the multi-dimensional and high-precision detection mode is beneficial to timely finding potential defects of the UV finish paint, ensuring the quality of the UV finish paint, improving the overall quality and reliability of a product using the UV finish paint, reducing product problems caused by poor finish paint performance, and improving the product quality. And meanwhile, the quality control cost in the production process is reduced.

The invention discloses an image description generation system, a training method, a generation method and electronic equipment, and belongs to the technical field of image description. According to the method, visual features of an image are mapped to a visual and language comparable space, after a semantic information sequence is obtained, cross-modal semantic calculation of the semantic information sequence and the visual feature sequence is achieved through a Transform decoder, the middle hidden state of each candidate vocabulary is obtained, and then a corresponding directed acyclic graph is constructed; and after an optimal path is selected from the directed acyclic graph, the optimal path is directly mapped into image text description by a linear classifier. According to the method, visual information of the image and semantic information contained in the image are fully utilized, the sequential relation between words is learned by introducing the directed acyclic graph, the smoothness of description generation is improved, the method has non-autoregressive decoding attributes, and high-quality image text description can be generated at a high speed.

The invention provides an image text description generation method, electronic equipment and a readable storage medium. According to the method, the object perception prototype learning module and the global context feature extraction module are introduced, so that fine-grained information and global semantic understanding in the image are effectively balanced. The visual backbone network module can extract multi-scale and multi-level image features and perform fusion, thereby enhancing the expression ability of the image features. The object perception prototype learning module further extracts an object prototype from the fusion features to ensure that the model can accurately capture key objects and attributes thereof in the image, and the global context feature extraction module ensures that the overall context of the image is fully understood. On the basis, the encoding and decoding module combines the global context and the object prototype to generate the text description, so that the semantic splitting phenomenon in the traditional method is avoided, and the detail information in the image is effectively reserved, thereby improving the accuracy and integrity of the image description.

The invention provides a vehicle re-identification data set generation method and device based on a diffusion model, and belongs to the technical field of vehicle re-identification, and the method comprises the steps: building a cue word template based on vehicle attributes, and carrying out the iterative optimization; taking a real vehicle re-identification unmarked image as input, predicting an image-text pair of an unmarked vehicle by utilizing a pre-trained multi-modal image description model, and performing fine adjustment on a diffusion model; and based on the optimized cue word template and the fine-tuned diffusion model, performing optimization in combination with target detection and semantic filtering, constructing a vehicle re-identification data set generation model, and generating a vehicle re-identification data set. According to the method, the effect better than that of an existing virtual data set generation method can be achieved on a downstream vehicle re-identification data set, the number of real data needed by the generation method is only 1% of that needed by an existing method, the real data does not need to be labeled, privacy protection is facilitated, and data labeling expenditure is reduced.

The invention discloses a unified streaming processing method, system and device for multi-modal AI interactive content, a medium and a program product, and the method comprises the steps: receiving a request which is sent by a user and comprises an image and a text, and constructing a request context; extracting image features; carrying out image analysis and outputting an image description text; performing intention recognition and outputting an intention; multi-stage reasoning is carried out, and thinking content fragments are generated in a streaming mode; buffering and releasing thinking content fragments; generating text content fragments based on multi-stage reasoning; buffering and releasing the text content fragments; performing recommendation triggering based on the image description text and the intention; recommending content fragments; inserting a recommended position, buffering and issuing; and process event management: issuing event notifications when the beginning, any step fails and the end. The method is a universal method capable of transmitting different types of outputs in a single ordered stream, the analysis cost can be reduced, and the interaction experience and expansibility are improved.

Computer-implemented methods, systems, and devices for generating security challenges are described. An identity management system may obtain image descriptions. The image descriptions may include a first image description set that corresponds to a sequence of events and a second image description set that is unassociated with the sequence of events. The identity management system may obtain images based on the image descriptions. The images may include a first image set that corresponds to the sequence of events and a second image set that is unassociated with the sequence of events. The identity management system may generate a security challenge using the images. The security challenge may request for a user to identify the sequence of events from the images. Identification of the sequence of events may be based on each image of the first image set being contextually relevant to the sequence of events.

The invention relates to the technical field of data processing, and provides an ultra-high voltage circuit breaker image data set construction method, device and equipment and a storage medium, and the method comprises the steps: obtaining multi-angle and multi-state images of an ultra-high voltage circuit breaker and an assembly thereof, and carrying out the preprocessing, and obtaining a processed image; generating an image description corresponding to each processed image; carrying out fine labeling on the calibrated ultra-high voltage circuit breaker instance in each processed image; creating corresponding positive and negative sample pairs according to the image description; performing supplementary labeling and scene expansion on the positive and negative sample pairs according to the multi-target scene to obtain labeled and expanded positive and negative sample pairs; and constructing an ultra-high voltage circuit breaker data set according to the image description, the annotation and the expanded positive and negative sample pairs. Through the technical scheme, the learning effect of the model in the training process and the recognition accuracy during testing are ensured, and the understanding of the model for abstract concepts and missing situations and the ability of coping with complex scenes are improved.