309 results about "Video retrieval" patented technology

The invention discloses an automatic labeling and acquisition standardization method and system based on video content, and the method comprises the steps: decoding a video file into a processable video frame sequence, carrying out the standardization processing, unifying the format and the coding standard, and adjusting the resolution and the frame rate; performing content analysis on the standardized video frame sequence, extracting key frames, identifying objects, scenes and actions, and generating a video analysis result and a preliminary label; according to a video content analysis result, separating an audio stream from the video to carry out noise reduction processing and voice-to-text analysis, and combining the processed text with the video content analysis result to generate multi-modal comprehensive annotation information; based on the comprehensive labeling information, taking feedback of the user in video retrieval, label clicking and watching behaviors as an adjustment basis of label weights, and updating weight distribution of labels by analyzing user behavior data; labeling information and weight distribution are integrated into an index database, and indexes are updated in real time according to user behaviors and content changes.

Methods are provided for generating distilled multimedia data sets tailored to user's persona and / or task(s) to be performed associated with an enterprise network and enable interactive contextual learning using a multi-modal knowledge graph. Methods involve obtaining multimedia data from one or more data sources related to operation or configuration of an enterprise network and determining context for generating a distilled multimedia data set based on at least one of user input and user persona. The methods further involve generating, based on the context, the distilled multimedia data set that includes a set of multimedia slices generated from the multimedia data using a multi-modal knowledge graph. The multi-modal knowledge graph is generated using a graph neural network and indicates relationships among a plurality of slices of the multimedia data. The methods further involve providing the distilled multimedia data set for performing one or more actions associated with the enterprise network.

The invention relates to the technical field of intelligent information retrieval, in particular to an intelligent database video retrieval method based on a big data technology, which comprises the following steps: S1, carrying out space-time slicing processing on an original video stream; s2, extracting a visual feature vector, an audio waveform vector and a text description vector; s3, constructing a cross-modal incidence matrix; s4, constructing a layered mixed index structure which comprises a real-time updating layer and a static storage layer; s5, after a user retrieval request is received, candidate video set screening is carried out based on the layered mixed index structure, and a retrieval result is generated; and S6, updating the cross-modal incidence matrix, and synchronously updating the weight parameter of the layer. According to the method, the retrieval accuracy is improved through cross-modal feature fusion, the data storage and retrieval efficiency is optimized by adopting a hierarchical mixed index structure, and the index weight is dynamically adjusted in combination with user behavior feedback, so that efficient, accurate and intelligent video retrieval is realized.

The invention discloses a text video retrieval method based on human brain scene memory path inspiration. The method comprises the following steps: performing content representation extraction on target text data or target video data by utilizing a content coding component to obtain multi-scale target representation; performing context representation extraction on the target text data or the target video data by using a context coding component to obtain a target token; inputting the multi-scale target representation and the target token into a hyperbolic graph neural network to obtain a target scene representation; taking the target scene representation as a target index; calculating the similarity between the representation of the to-be-retrieved text or video and the target index, and screening the to-be-retrieved text or video according to the similarity to obtain a target retrieval result; according to the method, multi-level semantic features are comprehensively captured through multi-granularity information fusion, multi-modal and multi-granularity high-order information is fused through hyperbolic graph convolution operation, the complex relation between the text and the video can be better captured, and the precision and efficiency of text and video retrieval are remarkably improved.

The invention discloses a universal scene retrieval analysis method and system based on multi-modal feature fusion, the method comprises a video analysis step and an application service step, and the application service step comprises the steps of receiving a user input request, describing a multi-dimensional standardized video tag based on a video summary, and obtaining a multi-dimensional standardized video tag; the steps of cross-modal video retrieval, dynamic knowledge enhancement question answering and interactive enhancement analysis can be executed, efficient video preprocessing is achieved by constructing an offline feature library, the retrieval precision is improved by adopting a cross-modal feature fusion technology, the analysis authority is enhanced in combination with a dynamic knowledge base, and the interactive enhancement analysis is supported to achieve abnormal early warning. The method has the advantages that the offline video processing efficiency is improved, cross-modal feature fusion retrieval is realized, and the authority of an analysis result is enhanced.

Various examples, systems, and methods are disclosed relating to an agentic curation pipeline. One system can process questions and other inquiries about video content by using a combination of models and stored information. The system can receive a query related to an event in a video, selects relevant portions of the video using embeddings, and apply the selected video data and a related sub-query to a video model. The output from the video model can be used by a language model, along with stored context, to generate an answer to the original query. The system can returns the answer to the requester.

The invention relates to the field of artificial intelligence cross-modal retrieval, and provides a video text cross-modal retrieval method and system based on spatio-temporal feature fusion. The method comprises the following steps: carrying out key frame sampling and time sequence partitioning on an input video, extracting static visual features through a spatial feature network, and extracting motion features through a time dynamic network; a self-adaptive gating fusion module is adopted to dynamically calculate spatial-temporal feature weights and perform weighted fusion; extracting text semantic features by using a pre-training language model; constructing a double-flow projection network to map video fusion features and text features to a unified measurement space, and optimizing a feature distance by adopting a contrast loss function containing difficult negative sample mining and intra-modal constraint; and outputting a retrieval result according to the cosine similarity sequence. The system comprises four units, wherein the gating fusion module is integrated with an FPGA acceleration circuit. According to the method, mAP (at) 10 is equal to 0.78 in a UCF-101 data set, the time sequence action retrieval accuracy rate is 92.8%, and the single video retrieval delay is 23 milliseconds.

The invention relates to a micro-video popularity prediction system and method. The method comprises the following steps: constructing a micro-video data set; respectively training a micro-video retrieval module, a multi-mode processing module, a GRU4Rec module and a prediction module by utilizing the micro-video data set; a to-be-queried micro-video is obtained, on one hand, the to-be-queried micro-video is input to a micro-video retrieval module and a multi-modal processing module to be processed, and multi-modal features are obtained; on the other hand, the information is input to a GRU4Rec module to be processed, and recommendation features are obtained; and carrying out fusion processing on the obtained multi-modal features and the recommendation features to obtain a micro-video popularity prediction result corresponding to the micro-video to be queried. Compared with the prior art, the social network information and the recommendation system information except the video content are integrated, so that the factors influencing the popularity of the micro-video are more comprehensively captured, the influence of the user behavior on the popularity of the video is simulated, the time dynamic state of the event sequence can be captured, the understanding of the user behavior and the video propagation is enhanced, and the user experience is improved. And the accuracy of micro-video popularity prediction is greatly improved.

The invention discloses a multi-granularity video retrieval method and device based on a multi-modal large model, computer equipment and a readable storage medium, and the method comprises the steps: firstly obtaining video query information input by a user, carrying out the intention recognition to obtain a query field, rewriting the query information and the field to obtain a video query vector, and carrying out the retrieval of the video query vector; and retrieving in a preset retrieval video knowledge base according to the vector and the field, and finally obtaining the target retrieval video content, so as to improve the efficiency and accuracy of video retrieval and adapt to multi-field retrieval requirements.

The invention discloses a video retrieval generation method and device based on sparse representation and reordering, and relates to the technical field of cross-modal video retrieval. The method comprises the following steps: acquiring a query text, a candidate video sequence and a video retrieval generation model; obtaining a query text dense representation and a candidate video sequence dense representation according to the encoder module, the query text and the candidate video sequence; through a sparse representation module, text sparse representation is generated according to the query text dense representation, candidate video sparse representation is generated according to the candidate video sequence dense representation, and reverse index screening is performed on the candidate video sequence to obtain part of candidate videos; determining a global score of each video in the partial candidate videos through a cross attention module, and sorting the partial candidate videos; and through a generation module, generating a target text corresponding to the query text according to the sorted candidate videos and the query text. By adopting the method and the device, the retrieval efficiency is improved in a sparse representation and reordering mode.

The embodiment of the invention relates to a training method of a video retrieval model and a video retrieval method and device, and the method comprises the steps: obtaining a video sample and a retrieval text sample, obtaining a visual feature, a subtitle text feature and an audio feature corresponding to the video sample, configuring initial weight parameters corresponding to the visual feature, the text feature and the audio feature, performing feature fusion on the visual features, the subtitle text features and the audio features based on the initial weight parameters to obtain video features, and adjusting model parameters and the initial weight parameters of a to-be-trained video retrieval model when a loss value between the video features and retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold value, and obtaining a video retrieval model until the adjusted loss value is smaller than a preset threshold value. According to the video retrieval model obtained through training, the weight parameters corresponding to the visual features, the subtitle text features and the audio features can be accurately distributed, and therefore the retrieval result of video retrieval based on the video retrieval model is more accurate and comprehensive.

The invention discloses a fusion movie and television play content retrieval method based on a large language model, face recognition, target detection and a cross-modal vector. The method comprises the steps that videos are preprocessed and stored; performing word segmentation and optimization processing by using a large language model according to a retrieval text of the video content input by a user; performing OCR (optical character recognition) on the staff table of the video content and matching a character role association relationship; performing target detection on the object; for scenes and event contents involved in user retrieval, video frames are converted into vector representation through cross-modal mapping established by a CLIP model, the vector representation is compared with vector representation of a query text, and the similarity between the video frames and the query text is calculated; and finally, carrying out secondary sorting through the hit number and the final weighted score. The invention further discloses electronic equipment, a medium and a product. According to the method, the movie and television play media video material content retrieval capability is improved, comprehensive judgment is carried out through a fusion algorithm, and a high-quality video retrieval scheme is obtained.

The invention discloses a video retrieval method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring each current to-be-retrieved video; dividing each video to be retrieved into a plurality of split videos; for each target retrieval dimension, obtaining a text description of the target retrieval dimension of each split video; wherein the target retrieval dimension comprises a subtitle dimension, an audio dimension and a visual dimension; the text description of the visual dimension of one split video is combined by the context text description of the split video and the text description of each key frame of the split video; combining the text description of the target retrieval dimension of each split video with a retrieval problem, and performing retrieval enhancement to obtain a plurality of retrieval videos based on the target retrieval dimension; weighting the text description of each target retrieval dimension of each retrieval video to obtain a plurality of sub-mirror multi-dimensional text descriptions; and combining the multi-dimensional text description of each sub-mirror with a retrieval problem to carry out retrieval enhancement to obtain a final retrieval result.

The invention discloses a video ROI retrieval method and system based on a multi-mode neural network, and the method comprises the steps: carrying out the key frame extraction of an input video, and obtaining a key frame; performing visual feature extraction, audio feature extraction and text feature extraction on the key frame to obtain a multi-modal feature of the key frame; inputting the multi-modal features of the key frames and the query text into a multi-modal neural network, and outputting timestamps of candidate video frames containing the region of interest and diagonal coordinates of candidate frames; arranging the timestamps of the candidate video frames according to a time sequence, and determining at least one interception interval of the input video according to a difference value of the timestamps of the adjacent candidate video frames; and intercepting the input video by using the interception interval, generating an ROI frame according to the diagonal coordinates of the candidate frame, and covering the ROI frame on the corresponding video clip to obtain a video ROI retrieval result. The invention relates to the technical field of artificial intelligence, and solves the technical problem of insufficient retrieval accuracy of a video region of interest ROI in the prior art.

The invention relates to a text video retrieval method for perceiving importance of spatio-temporal characteristics, and belongs to the technical field of text video retrieval. The method specifically comprises the steps that feature extraction is conducted on each frame and text description in a video, key frames in the video are selected through an importance perception mechanism, and semantic information of a query text in the transmission process is kept through an information source channel joint coding method based on semantic communication. And aggregating image block features of each frame into an object prototype, matching an object with phrases in text description, and sensing spatial information in a video frame. According to the method, different objects are combined into different event prototypes, the event prototypes are matched with the whole text description, the fine-grained matching between the text and the video is improved by utilizing the association time information between frames, and the content of the video is optimized by optimizing the spatial-temporal feature processing and feature importance evaluation of the video content. Efficient retrieval can be achieved in large-scale video data, redundant calculation is reduced, and video retrieval precision is improved.

The invention provides an image video retrieval method based on a domain fine-tuning large language model, which comprises the following steps: performing fine-tuning on a pre-training model to obtain a fine-tuning pre-training model for intention classification and keyword extraction; performing dynamic iteration screening on an optimal prompt template through Monte Carlo tree search in combination with a hidden Markov model (HMM); performing noise filtering on the keyword list, and predicting category labels of the filtered keywords through a conditional random field model to obtain a keyword enhancement set; combining with the user intention to generate a query condition, and obtaining a candidate resource set; and according to the similarity between the user query text and the candidate resource set, and in combination with the optimal prompt template, obtaining the resource path with the highest matching score between the user query and the candidate resource, and obtaining the retrieved image or video, so that the identification deviation possibly occurring when a general model processes proper nouns and terminologies can be effectively solved, and the user experience is improved. And the retrieval accuracy and response speed are improved, so that the retrieval accuracy and professional adaptability are improved.

The invention belongs to the field of video retrieval, and particularly discloses a video segment retrieval method and system based on a large model, a terminal and a medium, and the method comprises the steps: segmenting each piece of original video data into a plurality of video segments; the video segments are input into a pre-trained multi-modal large model, feature descriptions and target video frames of the video segments are extracted, and the feature descriptions comprise natural language feature descriptions and multi-modal feature vectors; combining the natural language feature description of each video segment, the multi-modal feature vector and the target video frame to obtain video segment feature representation; storing the video segment feature representation in a database; and obtaining retrieval information input by a user, comparing the retrieval information with each video segment feature representation in the database, screening out video data matched with the retrieval information, and feeding back the video data to the user. According to the method, the accuracy of video segment retrieval is improved by fusing the natural language feature description and the multi-modal feature vector and accurately extracting the target video frame.

The invention provides a video retrieval method and server based on semantic embedding and video memory coding, and the method comprises the steps: executing visual content analysis on continuous frames of an input video through a pre-trained visual language large model, and mapping the visual content of each frame into a semantic embedding vector; a video memory coding module is constructed based on the time sequence features of the semantic embedding vectors, and the continuous frame semantic embedding vectors are dynamically coded according to the time sequence to generate a video memory feature sequence containing time sequence association information; performing semantic analysis on the query text based on the visual language large model, and converting the query text into a query semantic embedding vector which is in the same spatial dimension as the semantic embedding vector; taking the query semantic embedding vector as a retrieval basis, performing similarity matching in a video memory feature sequence, and determining an associated video clip candidate set through feature association analysis in a time sequence window; and performing time sequence coherence verification on the candidate set, and outputting a matched video clip in combination with the semantic embedding vector time sequence correlation degree.

The invention provides a video clip retrieval method and device based on comparative learning, equipment and a medium, and relates to the technical field of video retrieval. According to the method, by adopting the pre-trained video clip retrieval model, the first text feature corresponding to the video retrieval text and the first video feature in the target video data can be efficiently extracted from the text and the video content, and the feature similarity between the features is calculated by applying the comparative learning strategy; and thus, related video clips can be accurately matched. According to the method, the dependence on computing resources is reduced, the resource consumption in the retrieval process is reduced, the retrieval speed and accuracy are improved, and the practicability and efficiency of video retrieval are enhanced.

The invention discloses a short video retrieval method combining a pre-training model and a dependency syntax tree, and relates to the technical field of multi-modal learning, computer vision and information retrieval cross fusion. The semantic information in the context is captured by using the advantages of BERT, and the deep semantic representation is generated, so that the meaning of the sentence is more accurately understood. A clear syntactic relationship is provided by using the dependency syntactic tree, and the structured dependency relationship between words can be revealed. The relationship can help the model to better understand logic semantics in sentences and extract text features related to tasks in a more targeted manner. Through the dependency syntax tree and the model, primary and secondary information in the long sentences can be distinguished more accurately, the problems of word order ambiguity and the like are eliminated, feature extraction is more stable, and robustness is improved. The dependency syntactic tree is explicit grammatical representation, and structured information of the dependency syntactic tree can assist in analyzing BERT implicit features, so that more explainable basis is provided for the decision process of the model, and the interpretability is remarkably enhanced.

The invention discloses a grid-coding-based video space-time retrieval method and device, the grid-coding-based video space-time retrieval method is executed in computing equipment, and the method comprises the following steps: mapping a coordinate range of a target area into a corresponding grid code set according to a Beidou grid standard on the basis of the coordinate range of the target area; acquiring video data with an intersection or inclusion relationship between the spatial range of the shot object and the target area; combining the shooting time of any video data with each grid code to generate a space-time grid code set; constructing an index structure corresponding to the video data by taking each time-space grid code in the time-space grid code set as a main key; and in response to query information of the user, analyzing a space-time range in the query of the user, mapping the space-time range into a corresponding space-time grid code range, executing retrieval based on the range, and returning hit video data. According to the method, rapid positioning and accurate retrieval of large-scale aerial photography video data are realized.

The invention discloses a text-video retrieval method and system based on multi-granularity learnable interaction, and relates to the field of data retrieval, and the method comprises the following steps: encoding a query text and an unpaired video in to-be-retrieved data, and extracting query text features and video features; obtaining a plurality of learnable vectors, carrying out cross-modal interaction on the learnable vectors and fine-grained video features, carrying out cross-grained alignment on query text features and learnable variables, carrying out fine-grained alignment on each word feature and the learnable vector, and aligning the query text features and video features; and capturing fine-granularity alignment information and coarse-granularity alignment information, capturing weights of different similar vectors among modals, summing multiple similarity scores based on the weights to obtain final multi-granularity similarity scores, and sorting the final multi-granularity similarity scores to obtain a final retrieval result. According to the method, cross-modal interaction and other operations are carried out through the plurality of learnable vectors and the fine-grained visual features, the fine-grained visual information is utilized, and the retrieval accuracy is improved.

A system includes a display, and a database including surgical videos, images of organs in a human body obtained from a medical imaging device, and images of disease in a human body obtained from the medical imaging device. A controller including a processor is coupled to memory, the database, and the display, and the memory stores information that when executed by the processor causes the system to perform operations. For example, the processor may determine first organ information from the images of the organs, and first disease information from the images of the disease. The processor my calculate a similarity score between the first organ information and the first disease information and second disease information and second organ information indexed to the surgical videos. The processor selects one or more of the surgical videos based on the similarity score, and displays the surgical videos on the display.

The invention relates to the technical field of machine learning, and discloses a video retrieval method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring video data and text data related to the video data; encoding video data by using a video encoder to obtain video encoded data, and encoding text data by using a first text encoder and a second text encoder to obtain a first text code and a second text code; according to the first text code, the second text code and the video code data, performing calculation by using a multi-modal model to obtain video clip correlation value data; obtaining a plurality of saliency vectors, and calculating to obtain a saliency mark according to the plurality of saliency vectors, the video clip correlation value data and the video coding data; and inputting the saliency mark and the video data into a retrieval model to obtain target video data. A multi-modal model technology is applied to video clip retrieval, so that a user can retrieve video clips conforming to text semantics, and creation is more efficient and convenient.

The embodiment of the invention provides a multi-mode combined video retrieval method and device. The method comprises the following steps: acquiring text information and visual information; extracting character features from the character information; extracting visual features from the visual information; extracting visual semantic features from the visual information according to the character features; extracting common features and difference features between the character features and the visual semantic features from the character features and the visual semantic features; querying a preset video information base according to the visual features and the common features to obtain a plurality of video retrieval results matched with the visual features and the common features; and screening the plurality of video retrieval results according to the difference characteristics to obtain a screened video retrieval result. According to the method, effective information of multi-modal information can be fused, the real intention of the user can be accurately understood, and the accuracy of multi-modal combined video retrieval is improved.

Provided is an efficient approach to establish a foundational video-text model for tasks including open-vocabulary video classification, text-to-video retrieval, video captioning and video question-answering. Some example implementations include a model which can be referred to as VideoCoCa. Example implementations reuse a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with little or minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules (for example, cross-frame attention layer or perceiver resampler) and finetune the modified architecture on video-text data, aspects of the present disclosure leverage findings that the generative attentional pooling and contrastive attentional pooling layers in the image-text CoCa design are instantly adaptable to “flattened frame embeddings”, yielding a strong zero-shot transfer baseline for many video-text tasks.

The invention relates to an intelligent video retrieval method and system. The method comprises the following steps: receiving a video retrieval keyword input by a user; historical video retrieval data of the user is called, and video retrieval behavior analysis is carried out on the user; according to the video retrieval behavior analysis result of the user, performing semantic analysis on the video retrieval keyword input by the user to obtain a natural language query statement; mapping the natural language query statement to a unified semantic space, and in the unified semantic space, calculating semantic similarity between the natural language query statement and the multi-modal fusion feature of each video to obtain a video similarity sorting result; historical video retrieval data of the user is called, the video similarity sorting result is adjusted according to historical feedback data of the user and the natural language query statement, and a video retrieval result is obtained; and outputting a video retrieval result.

The invention discloses a CNN-based video vectorization retrieval method, system and device and a medium, belongs to the technical field of artificial intelligence application, and aims to solve the technical problems of low video retrieval efficiency and low accuracy in the prior art. The technical scheme is as follows: constructing and training a CNN convolutional neural network model: constructing the CNN convolutional neural network model; inputting the channel data of each frame of picture into a CNN convolutional neural network model for training, and obtaining a trained CNN convolutional neural network model; video frame sampling and preprocessing: uniformly extracting picture frames from an original video, preprocessing each frame of picture, and obtaining the preprocessed picture frames; entity feature extraction and vectorization: performing deep analysis on each frame of image through a trained CNN convolutional neural network model, and capturing and extracting key visual features in the image; performing visual feature aggregation and normalization; and video storage and multi-mode retrieval.

The present disclosure generally relates to an integrated approach for retrieving biomedical information from clinical video presentations. In particular, the present disclosure is directed to video retrieval systems and methods of text-video retrieval from clinical video presentations.

The invention belongs to the technical field of surveillance video retrieval, and particularly relates to a pedestrian re-identification method and device and a medium. The method takes balance performance and efficiency as a core, a two-stage dynamic retrieval mechanism is provided, the two-stage dynamic retrieval mechanism comprises a coarse reasoning stage and a fine reasoning stage, and the difficulty degree of image query is judged through an early leaving strategy to realize on-demand distribution of computing resources: for simple query, global features are extracted only in the coarse reasoning stage for quick retrieval, and the calculation resources are extracted only in the fine reasoning stage; terminating subsequent reasoning; and for the difficult query, sending to a fine reasoning stage, and further extracting the features of the fine-grained part so as to carry out fine retrieval. In addition, in order to further reduce the calculation cost, the method also introduces a hybrid expert model to dynamically activate the network module to participate in calculation. By applying the method provided by the invention, the calculation cost can be effectively reduced and the retrieval efficiency can be improved while the retrieval accuracy is ensured.