282 results about "Image vector" patented technology

The invention discloses an embedded intelligent visual language large model knowledge base construction and application method and device, a medium and a product, and relates to the field of data intelligent processing. The method comprises the steps that an image encoder and a text encoder are adopted to encode image data and text data into vectors respectively, and image vectors and text vectors are obtained; fusing the image vector and the text vector, and generating a knowledge graph based on a fusion result to obtain an embedded intelligent visual language large model knowledge base; obtaining a user problem, and analyzing the user problem to obtain an analysis result; retrieving in a multi-modal knowledge base based on the analysis result to obtain a plurality of objects and a graph retrieval result; and combining the selected objects with the corresponding graph retrieval results to obtain triple information, converting the triple information into position text descriptions of the objects, inputting the position text descriptions into the multi-modal large language model, and outputting answers corresponding to the user questions. According to the method, the answer generation accuracy can be improved, and illusion phenomena are reduced.

A method relating to determining similarities between media, and systems and computer-readable media used to implement said method, includes determining an input image for comparison with a plurality of stored images in a media database; vectorising the input image to determine an input image vector representation; and comparing the input image with each of the plurality of stored images in a media database. The method further comprises outputting a set of direct matches and a set of exact matches as a detected comparison output for the input image. Further embodiments also relate to systems and methods for associating digital media with provenance information and creating verified copyright assets for attributing copyright ownership using the systems and methods disclosed herein.

The invention discloses an image cross-modal retrieval method and device based on a large language model and a medium, and the method comprises the steps: generating a text description corresponding to an original image through a BLIP model, and converting the text description into a corresponding image vector; establishing a mapping relationship between the image vector and the original image, and storing the mapping relationship and the image vector to a preset vector database; obtaining an image query description submitted by a user, optimizing the image query description through a preset language large model, and converting the optimized image query description into a corresponding query vector; determining a plurality of to-be-selected images corresponding to the image query description according to a first similarity between each image vector and the query vector in a vector database; and calculating a matching degree between the image query description and the plurality of to-be-selected images, and screening out a target image meeting the image query description from the plurality of to-be-selected images according to the matching degree.

The invention belongs to the technical field of artificial intelligence, and relates to an image-text question answering method, system and device based on multi-modal RAG and a storage medium, and the method comprises the following steps: 1) extracting multi-modal information from a PDF document, representing the multi-modal information as dense vectors, and storing the dense vectors in a text vector database and an image vector database; 2) obtaining a semantic embedding vector of the problem text and a multi-modal embedding vector of the problem text; obtaining a semantic embedding vector of the description text of the problem image and a multi-mode embedding vector of the problem image; 3) performing coarse screening in the text vector database and the image vector database by using the semantic embedding vector and the multi-modal embedding vector, finding out coarse screening text data and coarse screening image data, performing multi-modal fine ranking on the coarse screening text data and the coarse screening image data, and obtaining text data and image data which are retrieved and recalled; and 4) generating a final answer by the multi-modal large language model. According to the method, multi-modal data in a long document can be effectively analyzed, and information most relevant to a problem can be accurately retrieved.

The present disclosure relates to systems, non-transitory computer-readable media, and methods for selectively conditioning layers of a neural network and utilizing the neural network to generate a digital image. In particular, in some embodiments, the disclosed systems condition an upsampling layer of a neural network with an image vector representation of an image prompt. Additionally, in some embodiments, the disclosed systems condition an additional upsampling layer of the neural network with a text vector representation of a text prompt without the image vector representation of the image prompt. Moreover, in some embodiments, the disclosed systems generate, utilizing the neural network, a digital image from the image vector representation and the text vector representation.

The invention provides a vehicle retrieval question and answer method, a controller and a vehicle, and the method comprises the steps: carrying out the analysis of a vehicle specification document through a pre-trained layout analysis model, and obtaining target information; the target information comprises a text and a picture; constructing a text vector database and an image vector database according to the target information; constructing a question proposed by a user, and retrieving the constructed question based on the text vector database and the image vector database to obtain an initial answer; reasoning the initial answer by using a multi-modal large model to obtain a target answer; the target answer comprises a picture and text content; therefore, after the question asked by the user is obtained, in order to improve the retrieval quality, the question needs to be constructed, so that the question better fits the understanding of the vehicle; and then the constructed questions are retrieved in the text vector database and the image vector database at the same time, and text answers and image answers are given after the initial answers are reasoned, so that the accuracy of vehicle reply is improved, a user can conveniently understand related operation steps, and the intelligence of the vehicle is improved.

The invention provides a mathematical problem solving method and device based on a multi-modal large model and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a mathematical element image of a mathematical problem; inputting the mathematical element image into an image coding model to obtain an image vector output by the image coding model; the image coding model is obtained by training based on a sample mathematical element image and a positive sample text description and a negative sample text description corresponding to the sample mathematical element image; inputting the image vector into a self-adaptive module to obtain an image conversion coding vector output by the self-adaptive module; the adaptive module is obtained based on training of a sample image vector and a sample text vector; determining question stem characters of the mathematical question, and inputting the question stem characters and the image conversion coding vector into the large language model to obtain a prediction answering process output by the large language model; the large language model is obtained based on sample question stem characters, sample image conversion coding vectors and sample answering process training, and the mathematical problem solving capability of the multi-modal large model can be improved.

The invention relates to a small-sample fine-grained image classification method based on retrieval and a multi-modal large model, and belongs to the field of fine-grained image classification, and the method comprises the steps: obtaining all categories of fine-grained images, and setting a category prompt template in combination with the specific features of each category; arranging the fine-grained image data set; performing vectorization model coding on the picture and the category information to obtain an image vector and a text vector; performing retrieval recall on the candidate information; and obtaining a final fine-grained image classification result through a multi-modal large model by using a to-be-classified label-free image sample and a retrieved and recalled image set, text information set and category information set. According to the method provided by the invention, the cost of data set collection can be reduced, and the flexibility and expandability of a fine-grained image classification task are improved.

The invention provides a learning video recommendation method based on modal semantic space alignment, a medium and equipment, and relates to the technical field of multi-modal semantic space alignment, and the method comprises the steps: extracting a user behavior vector and vectors of different modes of a learning video, including text, image, audio and structure vectors; projecting the extracted vectors of different modes to a public semantic space by using a multi-layer perceptron, and in the public semantic space, performing vector splicing on the structure vector with a text vector, an image vector and an audio vector; performing modal comparison and modal matching on the spliced vectors, and obtaining a multi-modal feature vector of the learning video by optimizing semantic alignment between different modals; and fusing the multi-modal feature vectors, calculating the similarity between the user behavior vector and the fused multi-modal feature vector through cosine similarity, and carrying out sorting recommendation on all videos. According to the method, the modal semantic structure of the learning video can be fully understood, and accurate matching and recommendation are realized.

The present disclosure provides a method, a device, and a product for retrieval. The method includes acquiring context information related to an image and determining a representation of the image based on image data and the context information of the image, where the context information includes at least one of environment parameters, user behavior data, time elements, or field metadata. The method further includes encoding the representation as an image vector in a high-dimensional vector space and storing it into an image vector database. When retrieval is performed, a query that includes text information and that is for the image vector database is received, and an image associated with the text information is determined from the image vector database. The method according to the present disclosure can improve accuracy and efficiency for image retrieval.

The invention relates to the technical field of target recognition, solves the technical problem that few samples are low in accuracy in human activity recognition, and particularly relates to an indoor daily activity recognition method based on comparative learning and few-sample learning, which comprises the following steps: acquiring a time sequence data set of human activities; the time sequence data set is converted into two two-dimensional image vector matrixes containing data time information and dynamic characteristics, and data enhancement is carried out; initializing a few-sample comparison learning model; inputting the time sequence data without labels into a pre-training module; and inputting a part of time sequence data with a small amount of labels into a fine tuning module for activity identification. According to the method, a relatively good activity identification effect can still be obtained under the condition that a large amount of labeled data is not needed. The method has good robustness, can adapt to the complex environment of indoor daily activities, provides accurate human body activity recognition service for users, and is convenient and practical.

The invention provides an image compressed sensing joint reconstruction method and system, a storage medium and electronic equipment. The image compressed sensing joint reconstruction method comprises the following steps: acquiring an original image, and acquiring an image vector based on the original image; performing primary reconstruction and secondary reconstruction on the image vector to obtain an initial reconstruction image; and further performing image post-processing on the initial reconstructed image based on the fine tuning network to obtain a target reconstructed image. The method not only pays attention to observation in an ideal environment, but also considers the influence of noise and other degradation factors in a reconstruction process, so that the capability of accurately deducing image features and modes in a noisy environment of the model is remarkably improved, and the robustness of a reconstruction algorithm is enhanced; moreover, the quality of the initial reconstructed image is optimized, the reconstruction error is reduced, the target reconstructed image is closer to the original image, and the reconstruction accuracy and visual effect are improved.

The invention discloses a medical image report generation method based on multi-modal large model retrieval enhancement, which relates to the technical field of medical image processing and comprises an image vector model module, a vector database and similarity detection module, a multi-modal large model module, a dynamic retrieval module and a retrieval enhancement generation prompt project module. The multi-modal large model module comprises a multi-modal input splicing unit, a reasoning information processing unit and a diagnosis report set generation unit, the dynamic retrieval module comprises a similarity threshold value analysis unit and a retrieval quantity analysis unit, and the similarity threshold value analysis unit is used for analyzing a maximum similarity value between a new image and a database. According to the invention, an image vectorization processing technology based on a CLIP-ViT model is adopted, so that the system can accurately capture fine features of a focus, and the risk of key information omission is reduced; a dynamic threshold adjustment algorithm is introduced, and the rigidity defect of a traditional fixed retrieval strategy is overcome.

This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that utilize a local implicit image function neural network to perform image segmentation with a continuous class label probability distribution. For example, the disclosed systems utilize a local-implicit-image-function (LIIF) network to learn a mapping from an image to its semantic label space. In some instances, the disclosed systems utilize an image encoder to generate an image vector representation from an image. Subsequently, in one or more implementations, the disclosed systems utilize the image vector representation with a LIIF network decoder that generates a continuous probability distribution in a label space for the image to create a semantic segmentation mask for the image. Moreover, in some embodiments, the disclosed systems utilize the LIIF-based segmentation network to generate segmentation masks at different resolutions without changes in an input resolution of the segmentation network.

The invention discloses a building plane element recognition method based on a graph neural network and a convolutional neural network, and belongs to the technical field of building plane element recognition. The identification method comprises the following steps: inputting a building plane pixel image, and carrying out preprocessing through image layer screening and information extraction to obtain a binary pixel image; vectorizing the binarized pixel image to obtain a vectorized image; performing image segmentation on the vectorized image; using the segmented vectorized image to construct a regional adjacency graph; performing regional adjacency graph optimization on the regional adjacency graph through graph analysis and regional merging by using a graph neural network, and identifying a room edge; calculating the area of the room; using the convolutional neural network image recognition model to recognize articles in the room; and performing room function prediction according to an identification result. According to the method, the problem of shape blurring caused by convolution in a traditional method is avoided. The graph neural network model training effect is improved, and the classification accuracy is improved. The method is suitable for various planar graph styles and is high in universality.

The invention discloses a Chinese sentiment classification method and device based on multiple modes and electronic equipment. The method comprises the steps of obtaining a target Chinese text; determining attribute words of the target Chinese text, and constructing a dependency font graph of the attribute words based on the attribute words; vectorizing the target Chinese text and the dependency font graph to obtain a text vector matrix and an image vector matrix; respectively determining attention scores of the text vector matrix and the image vector matrix; the emotion polarity of the target Chinese text is determined based on the attention score, the text vectors in the text vector matrix and the image vectors in the image vector matrix, and the emotion polarity comprises positive emotion or negative emotion. The Chinese emotion prediction method and device solve the technical problems that an existing Chinese emotion analysis method is single in analysis dimension, character pattern information of Chinese characters is not fully fused, only text information is considered, and consequently Chinese emotion prediction accuracy is low.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as financial science and technology and medical health, and discloses an identification method, device and equipment based on semantic space consistency, and a medium, and the method comprises the steps: obtaining and preprocessing a target image, a positioning text region and an object region, and generating region positioning information; identifying and correcting the text region to generate a target text vector, extracting an object region image feature and normalizing to generate a target image vector, and mapping the two to the same semantic space for consistency analysis to generate a fusion confidence coefficient and a preliminary identification result; and retrieving, comparing and outputting final identification information and the fusion confidence coefficient in a preset knowledge base based on the fusion confidence coefficient and the preliminary identification result. According to the method, complementary fusion of the text and the image is realized through semantic space mapping, and standardized verification is completed by combining with knowledge base comparison, so that the problem that a single recognition mode is easy to make mistakes is effectively solved, and the accuracy and reliability of a result are improved.

The invention relates to the technical field of images, in particular to an image vectorization method and system based on deep learning and edge detection, and the method comprises the steps: obtaining image data of a household appliance module, and carrying out the preprocessing of the obtained image data; taking the preprocessed image data as input to construct an edge detection model, performing optimization processing on an edge probability graph output by the edge detection model, performing edge detection on the preprocessed image data in sequence to generate multiple groups of edge graphs, and performing average processing on the multiple groups of edge graphs to obtain a comprehensive edge image; the optimized marginal probability graph and the comprehensive marginal image serve as input, marginal information is converted into a vector graph, a DXF file is obtained, the optimized marginal probability graph and the comprehensive marginal image serve as input, vectorization conversion is conducted through the Potrace algorithm, optimization processing on the marginal detection result in the early stage is combined, and the edge detection efficiency is improved. Vector graphs with higher precision and less noise can be generated.

The invention discloses an image generation method and device, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a first image and style information, wherein the first image is an image drawn by a user; converting the first image into a first image vector, and converting the style information into a style vector; mapping the first image vector and the style vector into feature vectors of the same dimension, and performing fusion processing to obtain a fusion vector; and generating a target image based on the fusion vector.

The invention provides a water and soil loss automatic monitoring and early warning method based on transfer learning, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, mining a project monitoring data vector and a project prediction data vector according to water and soil loss monitoring data; secondly, performing A-stage interference applying operation on a target remote sensing image of the target region based on the target interference vector to form a region remote sensing image vector; then, on the basis of a project monitoring data vector and a project prediction data vector, performing A-stage interference removal operation on the regional remote sensing image vector to form a multi-modal water and soil loss vector; and finally, performing water and soil loss prediction operation based on the multi-modal water and soil loss vector to form water and soil loss prediction data, and performing water and soil loss early warning operation based on the water and soil loss prediction data. On the basis of the content, the problem that in the prior art, the reliability of automatic monitoring and early warning of water and soil loss is relatively low can be solved.

The invention relates to the technical field of medical information, in particular to a multi-source data health intervention method, which comprises the following steps: firstly, acquiring texts, vital signs, image vectors and wearable signals, generating a unified vector through multi-modal coding, and extracting a topological life vector; constructing a spiking neuro-causal diagram based on a unified vector, and injecting a life vector as an external field into a Sheng differential equation to generate a dynamic pathological manifold; compressing the manifold into a tensor network state, constructing a quantum optimization model in combination with a causal diagram, minimizing energy expectation and topological risk to obtain an optimal intervention sequence, and reinjecting a target gradient to adjust the manifold in real time; after the sequence is subjected to clinical logic verification, a commitment value and a zero-knowledge proof are generated and written into a Byzantine account book, and meanwhile a control instruction is issued to an execution terminal. According to the method, millisecond-level decision making is achieved, the false alarm rate is reduced, and chain traceability is provided.

The invention belongs to the technical field of computers, and discloses an extraction system based on a highlight video in the automobile field, and the system comprises a text semantic analysis module which is used for obtaining the semantic information of the whole video. After a video is input, multi-modal information in an image and original information corresponding to the image and voice are extracted by framing the video, and then a text semantic vector is obtained by converting related titles, texts, voice and subtitles in the video; through a general classification model, a general quality scoring model, a general face detection and recognition model and a fine-grained detection model in the automobile field, image semantic vectors of the whole video can be formed based on the characteristics of the automobile field, and finally audio data in the video are obtained, feature vectors of the audio data are extracted and merged with the image vectors, and the video quality is improved. The semantic vector of the video is obtained, and a Cross Attention structure is utilized to enable the semantic vector of the text to guide the semantic features of the video for prediction.

The invention relates to the technical field of image detection, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a Chinese character font generation method, device, equipment and medium, the method comprises the following steps: obtaining a Chinese character image to be processed, and carrying out image coding on the Chinese character image to be processed to obtain a Chinese character image sequence; performing style feature extraction and semantic feature extraction on each Chinese character image vector in the Chinese character image sequence to obtain corresponding Chinese character style features and Chinese character semantic features; constructing a corresponding style loss function and a semantic loss function according to the Chinese character style features and the Chinese character semantic features; performing loss weighting calculation on the style loss function and the semantic loss function to obtain a target loss function; and performing font style generation on the to-be-processed Chinese character image according to the target loss function to obtain a target Chinese character font of the to-be-processed Chinese character image. The Chinese character font generation efficiency and font generation diversity can be improved.

Provided are a computer program product, system, and method for training and using a vector encoder to determine vectors for sub-images of text in an image to subject to optical character recognition. A vector encoder is trained to encode images representing text into vectors in a vector space. Vectors of images representing similar text have a high degree of cohesion in the vector space. Vectors of images representing dissimilar text have a low degree of cohesion in the vector space. An input image is processed to determine sub-images of the input image that bound text represented in the input image. The sub-images are inputted to the vector encoder to output sub-image vectors. The vector encoder generates a search vector for search text. Optical character recognition is applied to at least one region of the input image including the sub-images having sub-image vectors matching the search vector.

The invention discloses a multi-modal large model training method and an image-text translation method and device, and the method comprises the steps: determining a training sample comprising a training text, a translation text, a training image and inquiry information, training a to-be-trained multi-modal large model based on the training sample, and obtaining a trained multi-modal large model, the multi-modal large model comprises an image encoder, an adapter and a large language model, and the image encoder is used for extracting information with different granularities from pixel representation of an image and converting the information into image vector representation; the adapter is used for converting the dimension represented by the first image vector and the dimension represented by the second image vector into dimensions accepted by a large language model, and the large language model is used for outputting corresponding answers according to the input image vector representation and inquiry information. According to the invention, the fine-grained image encoder is added to the model framework, so that the defect that the multi-modal large model is insufficient in fine-grained image information encoding is overcome, and the visual capture capability of the multi-modal large model is improved.

The invention discloses an image vectorization method and device based on a neural diffusion curve. According to the method, geometric control point parameters of multiple sections of Bezier curves are predicted from end to end in an input grating image through a deep neural network, and the main contour and color structure of the image are accurately represented in a Bezier curve form; color control points on the two sides of the curve are automatically extracted through a micro color sampling module, and physically consistent diffusion curve representation is constructed; and performing neural approximation on the resolving process of the diffusion equation by adopting a rendering network based on a Fourier neural operator to realize end-to-end micro-reconstruction of the image generated by the prediction curve, thereby reversely optimizing the geometric prediction of the curve by using a reconstruction error. According to the method, manual intervention is not needed, image vector representation which is high in fidelity, infinite in scaling, lossless in editing and capable of supporting natural layering can be automatically generated while physical rendering consistency is kept, precision and efficiency are improved, and the method is suitable for the fields of image vectorization, digital art creation, micro rendering, graphic design and the like.

The invention discloses a multi-modal distribution network hot-line work knowledge retrieval method and system. The method comprises the following steps: acquiring target distribution network hot-line work data; constructing an initial vector embedding model based on a Chine CLIP model, and training the initial vector embedding model based on the target distribution network live working data to obtain a target vector embedding model; inputting the target distribution network live working data into a target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vectors and the text vectors to obtain a knowledge retrieval database; and obtaining distribution network hot-line work query data, performing cross-modal retrieval on the distribution network hot-line work query data based on the knowledge retrieval database to obtain a cross-modal vector corresponding to the distribution network hot-line work query data, and converting the cross-modal vector into corresponding modal data. According to the invention, efficient joint query of multi-modal information can be realized.

The present invention proposes a learning video recommendation method, medium, and device based on modal semantic space alignment, which relates to the field of multimodal semantic space alignment technology. The method includes: extracting user behavior vectors and vectors of different modalities of learning videos, including text, image, audio, and structure vectors; using a multi-layer perceptron to project the extracted vectors of different modalities into a common semantic space, and within the common semantic space, vector splicing the structure vector with the text vector, image vector, and audio vector respectively; performing modal comparison and modal matching on the spliced ​​vectors, and obtaining a multimodal feature vector of the learning video by optimizing the semantic alignment between different modalities; fusing the multimodal feature vectors, calculating the similarity between the user behavior vector and the fused multimodal feature vector using cosine similarity, and ranking and recommending all videos. The present invention can fully understand the modal semantic structure of learning videos, achieving accurate matching and recommendation.

The invention discloses a cross-modal image classification method, device and equipment, a medium and a program product, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like. The method comprises the following steps: respectively carrying out dimension transformation on a trained image prompt vector corresponding to an image coding branch and a trained text prompt vector corresponding to a text coding branch in a pre-training model; fusing the trained image prompt vector with the transformed text prompt vector, cascading with the image vector of the to-be-classified image, and inputting to an image coding branch; fusing the trained text prompt vector with the transformed image prompt vector, cascading with the text vectors of the plurality of category description texts, and inputting to a text coding branch; and determining target probability distribution of the to-be-classified image on the plurality of category description texts according to the output of the two branches so as to classify the to-be-classified image. According to the scheme, the robustness and generalization of the model on a cross-modal image classification task can be improved.

The invention discloses a municipal facility intelligent inspection method and equipment based on image and point cloud matching. The method comprises the following steps: constructing a municipal facility database containing a multi-view image template and a point cloud grid model; a scene image and a point cloud are synchronously acquired through an unmanned vehicle carrying an optical camera, a laser radar and a GPS; respectively carrying out image target detection and point cloud target detection; a cross-modal target matching stage: calculating a 2D / 3D nearest neighbor vector of an image mask centroid and a point cloud cluster centroid, performing direction included angle matching on the point cloud vector and an image vector after a y coordinate of the point cloud vector is returned to zero, and verifying matching correctness through vector modulus length ratio mode distribution; and when a defect is detected in any mode, a maintenance work order containing a defect position and a GPS coordinate is automatically generated. According to the invention, bimodal independent detection and result fusion are realized, the sensor calibration requirement is avoided, and the inspection efficiency and accuracy are improved.