2416 results about "Key frame" patented technology

The invention is suitable for the technical field of video acquisition, and particularly relates to a video stream acquisition method based on deep learning, and the method comprises the steps: carrying out the video acquisition through a self-adaptive strategy, and obtaining real-time collected video data; performing parameter adjustment through edge calculation to obtain preprocessed video data; performing key frame selection on the preprocessed video data to obtain key frame data; performing inter-frame difference compression on the key frame data through the optimized variational auto-encoder model, and outputting video frame compression data; and when the original video needs to be recovered, analyzing the video compression data, reconstructing through the pre-trained neural network model to obtain a reconstructed video sequence, and generating a complete video according to the reconstructed video sequence. According to the method, the non-key frame is directly generated in the decoding process by using the deep learning model, so that the compression ratio can be remarkably improved, and the video playing continuity and the image quality, especially the performance in high-dynamic and complex scenes, can be improved.

A long video understanding method based on iterative hierarchical key frame selection comprises the following steps: firstly, carrying out hierarchical frame sampling on an input video to obtain hierarchical visual features, and then carrying out a key frame query module on the Nth-layer visual features of a video frame and an input problem text to obtain a key frame query result; video key frame query is carried out in combination with context semantics to generate query vectors, whether the query vectors are enough to generate answers to the input questions is predicted, if not, the query key frames are updated in the (N + 1) th layer of visual features with the higher frame rate, and the input question texts are combined with the context semantics to carry out iterative query of the key frames again; and inputting the inquired key frame and the instruction or the question into a multi-mode large model to generate an understanding result. According to the method, the video frames are queried and the associated frames are screened out based on the user input problem, so that the performance and efficiency of long video understanding are improved, and the method has wide application prospects in scenes such as entertainment, security and education.

The invention discloses a multi-agent-based automatic generation method, system and terminal for a micro-drama, and the method comprises the steps: obtaining a user creation theme, and carrying out the story line generation processing and the split processing to obtain a plurality of split scripts which comprise commentaries, visual information, shot information and atmosphere information; performing dubbing conversion processing and subtitle generation processing according to the commentary to obtain a target dubbing audio and a target video subtitle; performing key frame generation processing according to the visual information to obtain a target key frame image, and performing video generation processing with the shot information to obtain a target video clip; performing background music generation processing according to the atmosphere information to obtain target background music; and performing time axis alignment processing and splicing processing on the target dubbing audio, the target video subtitles, the target video clips and the target background music to obtain a target synthesized episode. According to the invention, the production efficiency and the output quality of the micro-movie can be effectively improved.

The invention discloses an automatic labeling and acquisition standardization method and system based on video content, and the method comprises the steps: decoding a video file into a processable video frame sequence, carrying out the standardization processing, unifying the format and the coding standard, and adjusting the resolution and the frame rate; performing content analysis on the standardized video frame sequence, extracting key frames, identifying objects, scenes and actions, and generating a video analysis result and a preliminary label; according to a video content analysis result, separating an audio stream from the video to carry out noise reduction processing and voice-to-text analysis, and combining the processed text with the video content analysis result to generate multi-modal comprehensive annotation information; based on the comprehensive labeling information, taking feedback of the user in video retrieval, label clicking and watching behaviors as an adjustment basis of label weights, and updating weight distribution of labels by analyzing user behavior data; labeling information and weight distribution are integrated into an index database, and indexes are updated in real time according to user behaviors and content changes.

The invention discloses a traffic accident identification method based on unsupervised pre-training and a multi-modal large model, and the method comprises the following steps: S1, collecting and sorting video data in a traffic monitoring scene, selecting a video clip without a traffic accident as a data sample, extracting a video key frame, carrying out the unsupervised pre-training of a mask auto-encoder, and carrying out the unsupervised pre-training of the mask auto-encoder; reconstructing a video clip which is shielded randomly; s2, calculating a reconstruction error between the original video and the reconstructed video to evaluate the reconstruction quality, and screening out key frame video clips with poor reconstruction quality; according to the invention, through unsupervised training of the mask auto-encoder, even under the condition of limited annotation data, more traffic accident-free segments can be filtered, video segments in which traffic anomalies possibly exist can be rapidly screened out, and by constructing fine-tuning training of multi-modal traffic data on a multi-modal large model, the real-time performance of the model is improved. The method can improve the recognition accuracy and efficiency of a traffic accident in a complex scene, provides real-time alarm and detailed analysis, and takes countermeasures in time.

The invention discloses a video content semantic understanding and text description generation method based on deep learning, and relates to the technical field of multimedia information processing.The method comprises the steps that the semantic similarity of a text and a video frame is calculated through a CLIP model, related key frames are selected, and features are aggregated; respectively extracting audio, visual and semantic features; aligning different modal features by using self-attention, unifying dimensions of the LSTM, and then splicing and fusing; attention weights are calculated at a video level, a frame level and a channel level, and key information expression is enhanced; swin Transform encodes fusion features, and LSTM (Long Short Term Memory) decodes step by step to generate natural language description; and a text-video index database is constructed, and rapid retrieval is realized based on semantic similarity. According to the method, the mapping relation between the video features and the natural language is learned end to end through the deep learning model, dependence on a fixed template can be eliminated, and semantic description with various sentence patterns and coherent logic is generated.

The invention discloses a real-time three-dimensional fusion reconstruction method for a ship comprehensive test scene, and the method comprises the steps: carrying out the preprocessing of a target object image data flow, converting an image coordinate system into a camera coordinate system, obtaining a sparse point cloud model, and constructing a voxel three-dimensional grid; the method comprises the following steps: constructing a scale space by using a Gaussian difference pyramid to detect key points of an image under different scales, calculating a gradient direction histogram around the key points and taking the gradient direction histogram as a descriptor, matching the descriptor by using an Euclidean distance, and estimating relative pose transformation between the images to complete a local matching process. Projecting and mapping the local frame block to a reference frame to form a key frame, and entering a global matching process; and extracting a surface grid of the three-dimensional model from the dense body grid by adopting a Marching Cubes algorithm, traversing each cube unit in the surface grid, and obtaining distance information from a vertex to a nearest surface based on the cube units, thereby generating a final three-dimensional model of the test target object.

The invention relates to a long video question and answer method based on large model agent multi-round reasoning, an initial key frame sequence is determined according to questions and options, confidence scores of the options are generated, and the confidence scores are used for prompting that a large language model can still obtain relatively reliable answers under the condition that input information is insufficient. The positioned key frames are input into a large model agent together with a text context in a grid splicing form to deduce and predict answers. And the large model agent carries out evaluation based on the currently provided information and the answer prediction process, and determines whether the next round of iteration needs to be carried out to supplement missing visual information or not. According to the self-adaptive multi-round reasoning question and answer, different numbers of video frames required by different types of questions are considered, so that the accuracy and the quality of video question and answer are effectively improved.

The invention discloses a battery production defect inspection method, system, medium and equipment, and relates to the technical field of battery defect inspection. The method comprises the following steps: acquiring a video stream corresponding to each battery production line, and extracting a key frame sequence corresponding to each video stream; performing fusion analysis on spatial features and time sequence features of the key frame sequences to obtain abnormal state features; performing multi-stage screening on each key frame sequence based on the abnormal state features to obtain a target key frame with defects; performing space-time mapping on the target key frame, and determining an associated station corresponding to the abnormal production line; and according to the process parameters of the associated stations, carrying out grading evaluation on the abnormal state features, and generating abnormal early warning information. By implementing the technical scheme provided by the invention, misjudgment or missed judgment of abnormal conditions on the battery production line can be effectively avoided, so that the production quality of battery products is improved.

The invention relates to a high-fidelity cloud rendering cluster scheduling method and system, and the method comprises the steps: receiving a rendering task, generating a multi-level priority queue according to the task complexity, timeliness and resource demand classification, and automatically optimizing a scheduling strategy; monitoring the resource state of the heterogeneous computing node in real time; allocating tasks by using preemptive and round-robin scheduling strategies, and adjusting and coping with resource fluctuation in combination with a dynamic code rate; the tasks are decomposed by adopting a spatial blocking and time framing strategy, and an execution sequence is controlled according to a topological sorting algorithm; abnormal nodes are identified through heartbeat detection, and affected subtasks are migrated through incremental task updating. According to the method, efficient resource allocation, scheduling algorithm and idle key frame scheme can be realized, the GPU directly outputs the video stream, the rendering speed is remarkably improved, resource waste and operation cost are reduced through a dynamic resource allocation mechanism, a fault-tolerant mechanism is provided, the task is ensured to be normally completed under the condition of node fault, and the task efficiency is improved. And distributed rendering and synthesis of large-scale high-resolution images are supported.

The invention relates to an early warning method and device based on a multi-modal large model, equipment and a medium, and belongs to the technical field of deep learning. The method comprises the steps of collecting a real-time video stream; receiving the real-time video stream, and performing frame extraction operation on the real-time video stream to obtain a key frame image; calling a classification model to classify the key frame images, and screening out to-be-analyzed images needing further early warning analysis; inputting the to-be-analyzed image into a multi-modal large model, understanding the content of the to-be-analyzed image to generate image description, and generating an early warning result of the to-be-analyzed image in combination with the information in the constructed knowledge base and the image description; and outputting the obtained early warning result in real time. According to the method, the classification model is adopted to perform preliminary screening on the images, the images which do not need early warning obviously are screened out in advance, and the images which need to be further screened enter the multi-mode large model to be analyzed, so that the cost of the system can be effectively reduced.

The invention relates to the technical field of image processing, and provides an image target tracking method and device based on deep learning, and the method comprises the steps: obtaining an original frame sequence of a target video, carrying out the adaptive frame screening of the original frame sequence, and obtaining a plurality of feature frame sequences, key frames and common frames, performing feature extraction and feature mapping on all the feature frame sequences to obtain target trajectory nodes, performing trajectory optimization calculation on the key frames and the common frames by using all the target trajectory nodes to obtain key frame trajectories and common frame trajectories, and inputting the key frame trajectories and the common frame trajectories into a preset deep learning model to perform trajectory reconstruction optimization so as to obtain a target trajectory; and obtaining a tracking target trajectory. By performing adaptive frame screening on the original frame sequence and generating the feature frame sequence, the key frame and the common frame, the data processing efficiency and the information extraction accuracy are improved, the accurate generation of the target motion track is realized, and the problems of reduced tracking accuracy and delayed system response in a complex scene are solved.

The invention discloses a dense visual scene reconstruction method and system, and relates to the field of scene reconstruction, and the method comprises the steps: constructing a dense visual scene reconstruction architecture comprising a 3D Gaussian body representation module, a camera tracking module, a key frame management module and a map updating module; and reconstructing a dense visual scene by adopting a dense visual scene reconstruction architecture according to the to-be-reconstructed picture data. According to the invention, high-quality scene reconstruction and rapid scene rendering can be realized under a complex illumination condition.

The invention relates to the technical field of image processing and mode recognition, in particular to a classification method and device based on user video data and a medium, and the method comprises the steps: collecting video data uploaded by a user, and extracting a plurality of key frames from the video data; using a video feature extraction network and an image-level spatial feature extraction network with time sequence modeling capability to respectively extract time sequence features between the key frames and spatial features of each key frame, and generating video visual features based on the time sequence features and the spatial features; collecting behavior data of the user in the process of watching the video data, and constructing a final user behavior embedding vector based on the behavior data; fusing the final user behavior embedding vector with the video visual features to generate joint feature representation; and inputting the joint feature representation into a pre-trained graph neural network classification model for classification, and outputting one or more semantic category labels to which the video data belong.

The invention provides a video fine-grained understanding labeling method and device based on a multi-modal large model, relates to the field of video text labeling, and aims to realize automatic video fine-grained understanding labeling. According to the method, fine-grained static information extraction is realized through a plurality of key frames of a target video clip; obtaining a plurality of candidate static fine-grained text labels according to the static fine-grained description information; improving the extraction process of dynamic information in the form of video questions and answers, and obtaining a plurality of candidate dynamic fine-grained text labels according to the answer to the target question and the dynamic fine-grained description information; utilizing an evaluation model to obtain a confidence score corresponding to each candidate static fine-grained text label and a confidence score corresponding to each candidate dynamic fine-grained text label; and finally, taking the plurality of candidate static fine-grained text annotations and the corresponding confidence scores thereof, and the candidate dynamic fine-grained text annotations and the corresponding confidence scores thereof as fine-grained annotation information of the target video clip.

The invention provides a media data fusion method and system, core voice switching equipment and a storage medium, and the method comprises the following steps: carrying out the feature extraction processing of multi-source heterogeneous audio data and video data, and obtaining an audio data stream, a video key frame and the semantic features of the data; performing synchronization processing on the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data; the method comprises the following steps: synchronizing audio and video data, extracting key features of the synchronized audio and video data, obtaining content correlation features in the audio and video data, performing feature selection and feature combination processing on the key features of the synchronized audio and video data, and performing content correlation maximization optimization on the content correlation features according to weights of the content correlation features. Obtaining an optimized audio and video feature set; and constructing a data mixed stream optimization model, and carrying out fusion mixed stream processing on the optimized audio and video feature set to obtain fused audio and video mixed stream data.

The invention relates to an introspection lifelong SLAM method and system based on visual language model hidden space representation, and the method comprises the steps: extracting a semantic tag based on an RGB-D image through a semantic encoder, and generating a scene map and a semantic topological graph based on the RGB-D image and the semantic tag; generating a dynamic mask based on the scene map, obtaining a dynamic mask coverage rate, and screening key frames with high static confidence values based on the coverage rate; calculating camera pose estimation corresponding to the key frame in real time, sampling the key frame to realize layering of the key frame, and performing layering rendering by using a NeRF model to obtain a virtual view; the hidden space difference degree of the virtual view and the corresponding real image is calculated, whether error introspection needs to be carried out or not is judged based on the hidden space difference degree, and the system is used for achieving the method. Compared with the prior art, the method has the advantages that open semantic reasoning of VLM, high-precision reconstruction of NeRF and real-time positioning of SLAM are combined, and positioning and mapping accuracy is improved.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as financial science and technology and medical health, and discloses a task instruction generation method, device and equipment based on cross-modal fusion, and a medium, and the method comprises the steps: carrying out the decoding and noise reduction of an input video, generating a frame sequence, and recognizing a plurality of key frames based on the inter-frame similarity; extracting spatial features of the key frames to form a sequence, and generating video spatial-temporal features in combination with time features; performing semantic preprocessing on the input text to obtain text semantic features, and acquiring motion sensor signals to obtain motion features; fusing the video spatio-temporal features, the text semantic features and the action features to generate fused features; and generating a perception vector based on the fusion feature and outputting a task instruction. According to the method, multi-modal fusion is realized through key frame extraction and space-time fusion mechanisms in combination with text semantic features and action features, and the perception expression ability and the task instruction generation accuracy are improved by using time sequence information and multi-source perception input of the video.

The invention belongs to the technical field of information propagation, and discloses a short video popularity prediction method based on multi-modal retrieval enhancement, which comprises the following steps: extracting key frames from a video to be predicted, and obtaining the image description of each key frame; cascading the original text information of the to-be-predicted video with the image description of the key frame of the to-be-predicted video to form a text prompt of the to-be-predicted video; inputting a text prompt of a to-be-predicted video into the large language model to obtain a retrieval vector of the to-be-predicted video; retrieving a plurality of related videos from the retrieval pool; performing feature extraction on the video to be predicted and the related video obtained by retrieval by using a pre-trained large model, and respectively extracting text and visual features; performing cross-modal feature interaction on the to-be-predicted video and the related video to obtain a corresponding mode, and performing interaction to obtain a retrieval enhancement vector; and predicting the popularity of the short video according to the modality of the video to be predicted, the modality of the related video and the retrieval enhancement vector. According to the method, retrieval enhancement is introduced, so that the prediction accuracy is greatly improved.

Implementations utilize a LLM to generate content responsive to a user query directed to a video and cause audio data for the generated content to be rendered as a response to the user query. Implementations extract a subset of frames from all frames of the video as key frame(s) for the video, and utilize a vision-language model in generating a natural language description for the key frame(s) of the video. A prompt can be generated based on a transcription of the user query and based on the natural language description for the key frame(s) of the video. The prompt is processed as input, using the LLM, to generate the content responsive to the user query directed to the video.

The invention discloses a posture feature recognition method based on a video stream, and the method comprises the steps: carrying out the preprocessing of a continuous video stream, obtaining video frame training data, extracting a key frame and an adjacent frame in each frame of image, constructing a feature extraction module for a human body region, and obtaining a global frame, performing local extraction on the human body area by using adjacent frames on the left side and the right side to obtain local frames, and constructing semantic association information for the global frame through time sequence continuity between the local adjacent frames and the current key frame; acquiring enhanced feature representation by adopting a conditional feature aggregation algorithm; obtaining attitude sequence data through the attitude detail features; the method comprises the following steps: establishing three-dimensional coordinates, adaptively extracting posture change data by adopting a human body motion decoupling model, predicting human body posture characteristics through a smooth optimization strategy, and introducing a cross attention mechanism to realize deep fusion of spatio-temporal characteristics, so that the understanding ability of the model to a complex action mode is enhanced; and the attitude expression capability of the model in a sheltered or fuzzy region is obviously improved.

The invention provides a 3D scene reconstruction method and system based on adaptive key frames, and relates to the technical field of computer vision, and the method comprises the steps: obtaining continuous multi-frame images and inertial measurement unit data of a current scene; acquiring camera trajectory data and offset data of each inertial measurement assembly based on the inertial measurement unit data; carrying out target feature extraction and target tracking on each frame in the multiple frames of images to obtain feature point data of the to-be-tracked target; determining a key frame based on a scene complexity index and a system resource utilization rate index of the current scene; performing visual inertia joint optimization based on all feature point data, camera trajectory data and offset data, and generating point cloud data containing each feature point; and generating 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, and rendering the distribution to obtain a first 3D rendered image of the current scene. According to the method, real-time 3D scene reconstruction can be carried out in a complex and changeable application scene.

The invention discloses a long-time pedestrian re-identification method based on dual-path cooperation and key frame guided reconstruction. The method comprises the steps of firstly collecting a pedestrian video to be recognized, and extracting a video feature sequence; space and time position coding is introduced into the video feature sequence; capturing local fine-grained dynamic features through a local dynamic feature capturing path, and modeling long-range time sequence association through a cross-frame global feature modeling path; then, dual-path feature complementation is realized through bidirectional gating interaction; further screening out key frames, and realizing feature reconstruction through a full-frame attention propagation mechanism; and finally fusing the dual-path fusion features, the key frame guide reconstruction features and the refined features to generate pedestrian identity features. And processing pedestrian identity features to obtain standardized feature vectors, performing similarity comparison on the standardized feature vectors and pedestrian features in an image library, and returning a matching list. According to the method, video time sequence information is fully utilized, and the problem of insufficient robustness caused by appearance change in long-time pedestrian re-identification is effectively solved.

The invention belongs to the technical field of video processing methods, and discloses a high-performance video reasoning segmentation method based on time sequence marking. The method comprises the following specific steps: carrying out time sequence mark coding by using a multi-mode large language model, carrying out time sequence dynamic aggregation on a coded special mark, and carrying out mask decoding and propagation by using an SAM2 based on a key frame selection strategy of the time sequence mark. According to the method, multiple module designs are combined, and the time sequence marks generated by the multi-modal large model are utilized, so that the perception capability of the model on spatial features and time dynamics is effectively improved.

The invention discloses an intelligent evaluation method and system for food safety, a medium and equipment, and relates to the technical field of food safety management. The method comprises the following steps: acquiring a monitoring video of a kitchen area and environmental parameters acquired by a sensor network; extracting a key frame sequence of the monitoring video, and identifying at least one key area corresponding to each key frame image in the key frame sequence; performing anomaly detection on the environmental parameters and each key area to obtain an abnormal event; and calculating a food safety total score of the kitchen area based on each abnormal event, and generating a food safety level of the kitchen area according to the food safety total score. By implementing the technical scheme provided by the invention, the accuracy and efficiency of kitchen food safety evaluation can be improved, and the food safety management effect of the kitchen can be effectively improved.

The invention provides a panoramic video three-dimensional point cloud sparse reconstruction method and device integrated with key frame screening, and the method comprises the steps: obtaining a panoramic video of a target scene, and carrying out the video quality evaluation of the panoramic video, and obtaining a video evaluation result; determining a target resolution of downsampling according to the video evaluation result; carrying out downsampling processing on the panoramic video based on the target resolution to obtain a downsampled second video; for the second video, dynamically adjusting a frame extraction interval based on a preset key frame selection mechanism, and extracting a frame index of the key frame from the second video according to the frame extraction interval to obtain a plurality of frame indexes; and obtaining a plurality of corresponding key frames in the panoramic video based on the plurality of frame indexes, and obtaining the three-dimensional scene sparse point cloud of the target scene based on the plurality of key frames, thereby solving the technical problem of low processing efficiency in the prior art.

The invention relates to the technical field of visual SLAM (Simultaneous Localization and Mapping), in particular to a YOLOv8-based point-line fusion visual SLAM method in an indoor dynamic scene, which comprises the following steps of: respectively transmitting an RGB (Red, Green, Blue) image and a depth image acquired by an RGB-D (Red, Green and Blue) camera to a front-end tracking thread and a dynamic object detection thread, extracting ORB (Object Request) feature point and line features, and obtaining a visual SLAM result; meanwhile, the dynamic object detection thread adopts YOLOv8-Seg to complete instance segmentation of the dynamic object, a dynamic mask is generated, ORB point features and line features falling in the dynamic mask are judged as dynamic features, and then the dynamic features are removed; then, multi-view geometric constraint is adopted to further filter and segment missing dynamic feature points outside the mask; after the dynamic point-line features are removed, key frames are finally output and transmitted to a mapping thread, and positioning and mapping are carried out. According to the invention, the positioning precision of the visual SLAM system can be effectively improved.

The invention discloses a cartoon video generation method and system based on a video script, and relates to the technical field of video synthesis, and the method comprises the steps: based on structured script data, combining a predefined lens rule base and a reinforcement learning model, determining the type, duration and lens operation effect of a lens, and generating a lens splitting sequence through a dynamic lens splitting automatic generation mechanism, based on the split mirror sequence, generating an animation style key frame image through a diffusion model, selecting action data matched with the emotion label through a predefined action library, generating a voice waveform matched with the emotion label through a text-to-voice model, selecting a background music audio matched with the emotion label through a music library, and generating a multi-modal content stream; according to the method, the lens type, the duration and the lens operation effect are dynamically optimized through the reinforcement learning model, the defects of a traditional lens scheduling method based on static rule mapping in time continuity and narrative continuity are overcome, and the narrative fluency and the dynamic adaptability of the lens division sequence are obviously improved.

The invention relates to the technical field of animation processing, in particular to an animation real-time capturing and repairing method and system, and the method comprises the following steps: calculating the pixel change rate between continuous animation frames based on the time sequence data of the animation, determining the change threshold values of key frames and non-key frames, dynamically adjusting image collection parameters according to the change threshold values, and carrying out the real-time capturing and repairing of the animation. And the frame difference before and after adjustment is compared and analyzed. According to the method, through analysis of the animation time sequence data and real-time calculation of the pixel change rate, the continuity between the animation frames is effectively monitored and real-time adjustment is performed, so that the accuracy and efficiency of animation production are improved, visual errors are corrected in real time in the production stage, the high-cost later-stage repair requirement is reduced, and the production efficiency is improved. The frame-by-frame real-time semantic segmentation and visual element identification further enhance the accurate control of the animation content, ensure the image quality of each frame, and ensure the consistency and visual effect of the animation by refining the image quality and synchronously repairing the frames.

The invention discloses a non-inductive identity recognition and real-time tracking method based on personnel in a video. The method comprises the following steps: S1, acquiring video stream data and extracting a key frame image; s2, key point coordinates are extracted, and a time sequence skeleton sequence is constructed; s3, decomposing gait, action and attitude features to generate standardized features; s4, inputting the standardized features into an improved multi-scale image convolutional neural network, extracting identity representation features, and generating a target identity code; s5, matching identity codes based on similarity measurement, calculating a consistency score, and establishing an identity tracking trajectory; s6, tracking is carried out in combination with identity codes and consistency scores, identity mapping is dynamically updated, and non-inductive recognition is achieved; and S7, updating a track in real time, adjusting a tracking state, and ensuring identity stability. According to the method, the improved graph convolutional network and the consistency measurement technology are combined, the identity recognition precision and tracking stability are improved, and cross-scene non-inductive identity recognition is achieved.