Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

755 results about "Film editing" patented technology

Film editing is both a creative and a technical part of the post-production process of filmmaking. The term is derived from the traditional process of working with film which increasingly involves the use of digital technology.

High-quality video content automatic generation method and related equipment

The invention relates to a high-quality video content automatic generation method and related equipment, and the method comprises the following steps: carrying out the multi-modal feature extraction of an input original video material, and obtaining a video feature vector set; performing content semantic analysis on the video feature vector set to obtain a video semantic map; performing narrative structure optimization on the video semantic map through a deep reinforcement learning technology to obtain an optimized video story line; performing multi-modal content generation based on the optimized video story line to obtain a candidate video clip set; and performing intelligent editing and synthesis on the candidate video clip set to obtain target high-quality video content, thereby solving the technical problem that the generated content is lack of depth and attraction due to the fact that many automatic attempts are limited to simple image or clip splicing and potential story clues in video materials cannot be deeply mined.
Owner:SHENZHEN CHUANGYUAN INTERACTIVE TECH CO LTD

AI intelligent short video generation method and system based on multi-agent collaboration

The invention relates to an AI intelligent short video generation method and system based on multi-agent collaboration. The method comprises the following steps: S1, input analysis and sub-shot script generation; s2, obtaining materials; s3, sub-shot video generation; step S4: editing and synthesizing; wherein the plurality of sub-shot video clips are edited and synthesized into a complete video according to a script sequence, and style fusion processing is performed on the whole video by using a picture style unification algorithm; step S5, background music matching; and S6, outputting the slices. Through the structured sub-shot script generation, intelligent material completion, multi-shot style fusion and music synchronization technology, a professional short video of multi-scene coherent narrative can be generated, the structured sub-shot script is automatically generated, and intelligent material completion, picture style unification and music rhythm and emotion expression synchronization are realized.
Owner:苏州日报社

Video editing method and related equipment

The invention discloses a video editing method and related equipment, and the method comprises the steps: receiving an editing intention of a user, carrying out the semantic analysis of an intention analysis model, and generating an editing copywriting, a style and a structured instruction; the edited copywriting is split according to the edited copywriting; performing cross-modal analysis on video materials by using a video understanding model trained based on an open-source multi-modal model, and accurately positioning sub-lens segments; secondly, editing the sub-lens segments through an editing model to generate a preliminary sheet, performing quantitative scoring through a video scoring model, and obtaining a scoring result according to a multi-dimensional index; and finally, the result is fed back to the editing model, and iterative adjustment is carried out until a final slice is produced. According to the method, the editing efficiency is improved, the manual operation time is shortened, the material adaptability is enhanced, the personalized requirement is met, the editing effect consistency is improved, the video content understanding is deepened, the editing effect is optimized, the technical threshold is reduced, the existing technical problems are solved comprehensively, and an efficient, intelligent and personalized editing scheme is provided.
Owner:GUANGZHOU QUYAN NETWORK TECH CO LTD

Short video intelligent editing method and system based on multi-modal analysis

The invention discloses a short video intelligent editing method and system based on multi-modal analysis, and relates to the technical field of video editing. The method is used for improving editing efficiency and visual experience and comprises the following steps: extracting lip motion features of a character, visual saliency features of a commodity and a voice emotion intensity value from a target short video stream to form multi-modal time sequence data; afterwards, the voice stream is recorded, a product keyword timestamp is extracted, the alignment degree is calculated through dynamic time warping in combination with a visual saliency peak value, and a preliminary editing point set is generated through weighted evaluation in combination with an emotional intensity value; constructing an editing decision optimization model based on deep reinforcement learning, taking the multi-modal features as state input, adjusting the retention probability of editing points through a joint reward function, and selecting an optimal transition mode; and the lip movement and voice synchronization error before and after the editing point and the emotional and visual continuity of the transition section are analyzed, the discontinuous region is smoothed, and the edited finished product is output, so that precise short video intelligent editing is realized.
Owner:ANHUI XINGBANG DIGITAL TECHNOLOGY GROUP CO LTD

Marketing commodity short video generation method and device based on AI, medium and product

The invention discloses an AI-based marketing commodity short video generation method and device, a medium and a product, and relates to the field of short video generation. The method comprises the following steps: constructing a short video industry knowledge base; obtaining question information input by a user; searching answers from a short video industry knowledge base according to the question information; obtaining a video selected from the answers by the user as a reference short video, and obtaining commodity information uploaded by the user; the commodity information comprises commodity selling points and commodity materials; retrieving a reference copywriting from the short video industry knowledge base according to the commodity selling point, taking the reference copywriting as knowledge supplementary information, generating a script copywriting according to the reference short video and the knowledge supplementary information, and generating picture information according to the script copywriting and the commodity material; and editing the picture information to obtain a short video of the commodity information. The output efficiency of the running amount materials can be improved.
Owner:DALI MIRACLE (HANGZHOU) TECHNOLOGY CO LTD

Automatic video editing method based on semantic analysis

The invention relates to an automatic video editing method based on semantic analysis, and belongs to the technical field of video editing. The method comprises the following steps: extracting anchor point information of text content in a streaming media file based on a knowledge graph, establishing a support relationship, and determining a main anchor point, a sub anchor point and a argument anchor point by combining the support relationship and the anchor point information; constructing a dynamic trigger mechanism, setting a verification direction of the dynamic trigger mechanism, determining a semantic extension direction of the anchor point information according to the verification direction, and verifying whether a precondition chain supporting the anchor point information exists or not; and setting a credibility scoring strategy, determining the credibility of the anchor point information according to the logic complexity, the language confidence and the text consistency, forming a sequence set of the anchor point information, and splicing to form a logically coherent video editing text result. According to the method, automatic and intelligent editing of the video is realized based on semantic analysis, and the editing efficiency and quality are remarkably improved.
Owner:BEIJING GONGXIN INTERNET TECHNOLOGY CO LTD

Motion vector restriction for out-of-frame boundary conditions

Aspects of the disclosure provide a method and an apparatus for video decoding. The apparatus includes processing circuitry that determines motion information of a current block. The motion information indicates one or more reference blocks of the current block. If a motion information constraint indicates that the one or more reference blocks are within picture boundaries of respective one or more reference pictures, the processing circuitry reconstructs the current block based on the one or more reference blocks. If a first motion vector (MV) points from a region in the current block to a first reference region of a first reference block that is outside a picture boundary of a first reference picture, the processing circuitry determines a first clipped MV pointing from the region to an updated first reference region by clipping the first MV where the updated first reference region is within the picture boundary.
Owner:TENCENT AMERICA LLC

Short video clip synthesis method based on fuzzy logic

The invention discloses a short video editing synthesis method based on fuzzy logic. The method comprises the following steps: S1, carrying out digital video preprocessing on an input original short video; s2, forming a short video feature data set; s3, generating a fuzzy logic decision result; s4, performing multi-target and multi-parameter optimization on the fuzzy logic decision result by adopting a random score programming method to form an optimized editing parameter set; s5, generating a preliminarily synthesized short video; s6, generating a video quality evaluation result; and S7, carrying out iterative optimization on the preliminarily synthesized short video according to a video quality evaluation result, adjusting the editing parameters and re-executing the editing synthesis step until the video quality evaluation result meets a preset standard, and generating a final edited finished product. According to the method, the editing scheme can achieve the best balance in the aspects of visual rhythm, information integrity and emotion expression consistency.
Owner:BEIJING HESHENG ENVIRONMENTAL ENG TECH CO LTD

Multi-angle video automatic editing method based on content defect detection

The invention discloses a multi-angle video automatic editing method based on content defect detection, and relates to the technical field of video processing. According to the method, multi-track video materials are accurately synchronized to the same time axis through cross-correlation calculation, and flaws in video content are analyzed frame by frame and classified and identified by applying fuzzy detection of a dynamic threshold and combining jitter detection and shielding detection of rotation and displacement estimation. And according to a detection result, an editing scheme is intelligently generated through a dynamic programming algorithm of state transition optimization, high-quality fragments are selected, defective parts are automatically removed, the continuity and multi-view diversity of video contents are reserved, and finally, a high-quality recorded video is generated through integration. The video editing efficiency is greatly improved, the manual intervention requirement is reduced, meanwhile, it is ensured that the quality of the output video meets the professional application requirement, and the method is suitable for multiple scenes such as conference recording, educational training and multi-angle video production and has high practical value and popularization prospects.
Owner:SHENZHEN MSU-BIT UNIVERSITY

Video intelligent self-adaptive editing method and system based on deep learning

The invention provides an intelligent self-adaptive video editing method and system based on deep learning, and relates to the technical field of video processing.The method comprises the steps that firstly, a semantic mapping relation between a to-be-edited video material and a preset editing requirement is established, and an editing requirement mapping result is generated, the preset editing demand comprises a content style and a rhythm control demand, and then semantic feature association processing is carried out based on the mapping result to obtain a semantic association feature set comprising lens unit content semantic features and rhythm association features; then calling a pre-trained editing decision model (including a semantic matching module and a rhythm adjusting module) to carry out editing strategy matching on the set, generating a preliminary editing strategy set, generating an initial video editing scheme according to the preliminary editing strategy set, carrying out parameter adjustment on the initial scheme according to a strategy optimization suggestion output by the model, and carrying out video editing on the initial scheme; and a final video editing scheme is obtained, and intelligent self-adaptive editing of the video is realized.
Owner:WEIMAI TECH CO LTD

Method and device for generating video based on text

The embodiment of the invention provides a method and a device for generating a video based on a text, the method is applied to a server, and the method comprises the following steps: acquiring a plot description text uploaded by a client; splitting the plot description text to obtain a plurality of split texts, and determining a split prompt word corresponding to each split text; extracting at least one key content label and a feature text corresponding to each key content label from the plot description text, and determining a label prompt word corresponding to each feature text; and according to the split cue word and the label cue word, carrying out text generation processing to obtain a plurality of split illustrations, and according to the plurality of split illustrations, carrying out editing to generate a target video. According to the technical scheme, the process from splitting the plot description text and extracting the label to generating the illustration can be automatically completed, a user does not need to manually edit and process, the efficiency of converting the copywriting into the video is improved, and the user requirement of producing a large number of videos can be met.
Owner:SHANGHAI BILIBILI TECH CO LTD

Intelligent director and picture synthesis method of panoramic live broadcast system

The invention relates to the technical field of director and picture synthesis methods, in particular to an intelligent director and picture synthesis method of a panoramic live broadcast system, which comprises the following steps: acquiring panoramic video data shot by a panoramic camera; obtaining user feedback data; based on the panoramic video data, generating a shot selection strategy by using a deep reinforcement learning algorithm; detecting a live broadcast key event in real time by using a target detection and tracking algorithm based on the panoramic video data; generating a picture shooting plan sequence according to the shot selection strategy and the live broadcast key event; performing intelligent editing and synthesis on the panoramic video data based on the picture shooting plan sequence; and outputting the synthesized live video. And updating parameters of the deep reinforcement learning algorithm based on the user feedback data, thereby realizing full automation of the program directing process, greatly improving the efficiency, and enabling the system to continuously learn from live broadcast experience through the deep reinforcement learning algorithm and gradually improve the decision-making ability of the system.
Owner:DRAGONFLY INTELLIGENT VISION TECH (SHENZHEN) CO LTD

Video generation method and device and storage medium

The embodiment of the invention provides a video generation method and device and a storage medium, and the method comprises the steps: obtaining a video script, and obtaining a text feature of the video script; acquiring a video feature of each video material in the plurality of video materials; obtaining a similarity matrix between the text features and the video features; matching the video material with the video script based on the similarity matrix to obtain a video material sequence; and generating a target video file according to the video material sequence. According to the embodiment of the invention, through matching between the video script and the video material, retrieval and sorting of the video material are realized, so that video packaging is carried out based on the video material sequence, and automatic editing and production of the video are better realized.
Owner:BEIJING ZITIAO NETWORK TECH CO LTD +1

Video editing method and device

The embodiment of the invention provides a video editing method. The video editing method comprises the following steps: acquiring background music according to a video editing demand; on the basis of drumbeat recognition and emotion analysis on the background music by an expert template, selecting a key drumbeat for video picture switching, and generating an editing configuration file on the basis of the key drumbeat; segmenting and classifying the to-be-processed video through multi-modal analysis to obtain different types of video clips; and automatically selecting a video clip combination based on the editing configuration file, and adding a transition special effect to obtain an edited video. According to the embodiment of the invention, by introducing the intelligent editing scheme based on the expert template, the editing result can be ensured to be highly matched with the music rhythm, the scene segment selected in the editing process meets the preset emotion and theme requirements, and the logicality and ornamental value of the video are improved.
Owner:SHANGHAI HODE INFORMATION TECH CO LTD

Digital media content intelligent editing and special effect generating system based on artificial intelligence

The invention relates to the technical field of editing and special effect generation, in particular to a digital media content intelligent editing and special effect generation system based on artificial intelligence. Comprising a content understanding analysis module, an editing decision module, a special effect generation preview module, an audio processing module, a dynamic transition special effect module, a subtitle generation editing module, a virtual interaction module and a content distribution optimization module. The content understanding analysis module is used for video content identification and sentiment analysis; and the editing decision module is used for recommending an editing point and providing a dynamic editing strategy. The video content can be automatically analyzed, the editing point and the special effect adding position can be identified, the editing and special effect generation efficiency is greatly improved, the editing strategy and the special effect type can be dynamically adjusted according to the video content and the sentiment analysis result, the editing and the special effect can better meet the user requirement and the video content, and the user experience is improved. Abundant editing and special effect options are provided, and the editing and special effect generation process is simplified.
Owner:徐心冉

Intelligent video editing method fusing human face features and human voice features

The invention relates to an intelligent video editing method fusing human face features and human voice features, which comprises the steps of input preprocessing, multi-modal analysis, fusion scoring and automatic editing, and adopts a multi-modal mode for analysis, so that the recall rate of key segments is improved, and the efficiency of editing is improved. A personalized editing strategy is supported, facial expression changes and voice emotion peak values are aligned through dynamic time warping, a CLIP-like structure is used for training a bimodal encoder, the human face and the voice are mapped to a unified vector space, the association weight of the human face and the voice is automatically learned, and the intelligent degree of editing is improved.
Owner:BEIJING HEJUHUITONG E-COMMERCE CO LTD

Video processing method and video processing apparatus

A video processing method is provided. The video processing method obtaining a clip representation based on a frame index and a global representation of an input video, obtaining a residual frame representation based on the frame index and a residual representation of the input video using a residual neural network and outputting a final frame representation generated based on the clip representation and the residual frame representation.
Owner:SAMSUNG ELECTRONICS CO LTD +1

Video editing method and device, electronic equipment and nonvolatile storage medium

The invention discloses a video editing method and device, electronic equipment and a nonvolatile storage medium. The method comprises the following steps: acquiring an audio data stream of a target video, and segmenting the audio data stream into a plurality of audio clips; determining a classification result corresponding to the audio clip, and determining a time period corresponding to the audio clip as a candidate ball hitting time period when the classification result is that the audio clip contains the ball hitting sound; acquiring a video frame corresponding to the candidate ball hitting time period in the target video, and judging that the candidate ball hitting time period is a real ball hitting time period under the condition that the visual feature of the video frame is matched with a preset ball hitting rule; and determining an editing time point according to the real ball hitting time period, and editing the target video according to the editing time point to obtain a ball hitting round video clip. The technical problem that the accuracy of ball hitting detection segment detection is low due to the fact that ball hitting detection is conducted only through sound and is easily interfered by environmental noise in the prior art is solved.
Owner:CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD

System and method for generation and delivering of video clips of sport

A method, a system and a computer program product for delivering video clips of a sport to a user participating in the sport are provided. A camera is used to record and generate a video sequence of the sport participated by the user. A video clip is generated upon activation of a trigger by the user during participating the sport. The video clip includes at least a part of the video sequence and has a clip duration determined based on a timing of the activation of the trigger. The video clip can be delivered to the user upon request and / or payment. The trigger is provided in proximity to the user, such that the user can activate the trigger during participating the sport. The method, system and computer program product can be used to deliver video clips of sports, including table tennis, tennis, badminton, pickleball, Padel tennis, billiards, dancing, and so on.
Owner:PODPLAY TECHNOLOGIES INC

Video generation method and device based on multi-agent cooperation

The invention relates to the technical field of artificial intelligence content generation, and provides a video generation method and device based on multi-agent cooperation. The method comprises the following steps: selecting a proper copywriting processing agent according to text information input by a user, so as to use the copywriting processing agent to expand or condense a plurality of sub-mirror descriptions; selecting a proper image generation agent; inputting the picture description, the picture style and the mirror operation method in the mirror splitting description into the selected image generation agent to generate a mirror splitting picture; selecting a proper video generation agent; inputting the split picture and the character action and the mirror moving method in the corresponding split description into a selected video generation agent to generate a split video; and selecting a proper automatic editing agent, and performing post-production on the split video to generate a finished video. The technical problems that in the prior art, the video generation process is fragmented, and multi-element collaboration is difficult are solved, and automatic production of complex video content is achieved.
Owner:WUHAN SINAN YIYI INTELLIGENT TECHNOLOGY CO LTD

Accelerated video editing

A method for accelerated video editing that includes providing a video, segmenting, using a segmentation module, the video into a number of clips, and automatically computing, for each clip of the number of clips, corresponding metadata representing at least one attribute of the clip based on an analysis of contents of each clip of the plurality of clips. A profile script is generated based on the corresponding metadata, the profile script is generated to include one or more editing actions to be performed on a section of the video that corresponds to at least one identified clip from the number of clips. An editing tool is used to automatically edit the video based on the profile script to generate an edited video.
Owner:INTERNATIONAL BUSINESS MACHINE CORPORATION

Short play video editing method, system and device based on multiple modes and medium

The invention discloses a multi-mode-based short episode video editing method, system and device and a medium, and the method comprises the steps: carrying out the preprocessing of an original short episode video, and obtaining a second short episode video, subtitles and a subtitle timestamp; analyzing the subtitles, and generating a short episode abstract according to an analysis result of the subtitles; according to the short episode abstract and the highlight editing prompt, performing plot analysis and editing on the second short episode video, and performing editing to form a first video set; scoring videos in the first video set according to a multi-dimensional scoring rule, and screening out M highlight video clips before scoring; and adjusting the timestamps of the M highlight video clips according to the subtitle timestamps, and outputting the adjusted M highlight video clips. According to the method and the device, the full link from short video input to short play mixed video output is constructed, a user only needs to input the to-be-edited short play video, the mixed video can be automatically generated, and the video editing efficiency is improved while the video editing threshold is reduced.
Owner:GUANGZHOU TAIDONG TECH CO LTD

Video editing method and device, electronic equipment and storage medium

The invention discloses a video editing method and device, electronic equipment and a storage medium. The method comprises the following steps: segmenting a to-be-edited video to obtain a plurality of video clips; understanding text generation and subtitle generation are carried out on each video clip, and a clip understanding text and a clip subtitle of each video clip are obtained; determining a target description text based on fragment understanding texts and / or fragment subtitles corresponding to a plurality of target video fragments selected from the plurality of video fragments; and performing video synthesis based on the plurality of target video clips and the target description text to obtain a target video corresponding to the to-be-edited video. According to the method provided by the invention, the video editing efficiency is relatively high, and the video editing cost is relatively low.
Owner:SHENZHEN SIYUAN ELECTRONICS TECH CO LTD

Multi-agent collaborative short-play digital production system

The invention discloses a multi-agent collaborative short play digital production system, which belongs to the technical field of artificial intelligence, and comprises a data and state management module driven by a digital thread, which uniformly manages and synchronizes data, states and decisions of all production links through a throughout digital thread; the full-process intelligent production module takes an intelligent control center as a core, decomposes and dispatches a complex task to five professional intelligent agents for cooperative execution, and covers a complete production link from script generation, visual preview, video shooting and special effects to intelligent editing and final sound and picture synthesis; the man-machine collaborative evaluation and feedback closed-loop module provides an interactive interface for human experts to supervise, guide and correct key production nodes and records artificial decisions as feedback to form a closed loop of a continuous optimization system; according to the multi-agent collaborative short play digital production system provided by the invention, the digital thread penetrates through multi-agent collaboration, so that data tracing and state synchronization of the whole process of short play production can be realized, challenges such as narrative continuity, cross-modal consistency and physical authenticity are effectively solved, and the quality of generated content and the production efficiency are remarkably improved.
Owner:COMMUNICATION UNIVERSITY OF CHINA

Intelligent editing style learning and lens arrangement method and device

The invention discloses an intelligent editing style learning and shot arrangement method and device, and the method comprises the steps: firstly carrying out the shot segmentation of an original material video, obtaining a plurality of independent video clips, carrying out the multi-dimensional analysis of each video clip, generating a structured label text containing image semantic features and shooting grammar features, and carrying out the segmentation of the shot; semantic segmentation is carried out on a manuscript to be edited to obtain a plurality of text paragraphs, each text paragraph is matched with at least one edition example, finally prompt words are constructed and input into the generative language model to carry out shot arrangement, and the generative language model outputs a shot arrangement sequence of the current paragraph. The problem of error accumulation caused by the fact that a traditional intelligent editing method depends on an editing script intermediate link is solved, the defect that in the prior art, it is difficult to dynamically learn a specific editing style is overcome, collaborative optimization of semantic matching and editing grammar is achieved, and shot selection accuracy, arrangement fluency and style adaptability are remarkably improved.
Owner:CHENGDU SOBEY DIGITAL TECH CO LTD

Video processing method and device, equipment and storage medium

The invention provides a video processing method and device, equipment and a storage medium, and the method comprises the steps: segmenting a video material into a plurality of semantically coherent video clips, automatically extracting related video clips from the video material according to the text content provided by a user, and generating corresponding subtitles and voice broadcast ports matched with the related video clips, and finally synthesizing a section of finished video with clear logic and coherent content. According to the scheme, full-process automation from video material generation to final video file generation is realized, a large number of video materials can be quickly processed, the tedious manual editing process in traditional video production is reduced, the dependence on manual intervention is greatly reduced, the operation cost is reduced, the video production process is simplified, the video production efficiency is improved, and the user experience is improved. And the quality and the efficiency of finished products are also improved.
Owner:CTRIP TRAVEL NETWORK TECH SHANGHAI0

Video generation method and device, and storage medium

PCT designated stage expiredWO2025082445A1Selective content distributionEngineeringFilm editing
Embodiments of the present disclosure provide a video generation method and device, and a storage medium. The method comprises: acquiring a video script, and acquiring text features of the video script; acquiring video features of each video material among a plurality of video materials; acquiring a similarity matrix between the text features and the video features; matching the video material with the video script on the basis of the similarity matrix to obtain a video material sequence; and generating a target video file on the basis of the video material sequence. In the embodiments of the present disclosure, retrieval and sorting of the video materials are implemented by means of the matching between the video script and the video materials, and then video packaging is carried out on the basis of the video material sequence, thereby better achieving automated editing and production of videos.
Owner:BEIJING ZITIAO NETWORK TECH CO LTD +1

Editing processing method and device, electronic equipment and computer readable storage medium

The invention relates to an editing processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: displaying a video editing interface, the video editing interface comprises a material region and a timeline region, and when a material package option displayed in the material region is in a selected state, the timeline region is a timeline region; at least one material package identifier is also displayed in the material area, each material package identifier is used for indicating one material package resource, and each material package resource is a material package in a composite fragment form generated by a plurality of materials; in response to a target triggering operation on a target material package identifier in the plurality of material package identifiers, converting a target material package resource indicated by the target material package identifier into a target material package combination, and displaying the target material package combination in a timeline area; wherein the target material package combination is a combination of a plurality of target materials corresponding to the target material package identifier.
Owner:BEIJING ZITIAO NETWORK TECH CO LTD

AI automatic editing method and related equipment

The invention discloses an AI automatic editing method and related equipment, and the method comprises the steps: obtaining a video prototype template selected by a user, prompt information inputted by the user, and original materials uploaded by the user through a user interaction interface, and calling a language large model to generate a structured video script; performing association degree matching on the structured script and the original material through a target detection model, and screening the original material according to a matching result and a quality scoring model to obtain an original video clip; performing background music matching according to the video script to obtain background music, generating voice dubbing and a flower-character chartlet according to the video script, generating video subtitles according to the voice dubbing, performing time axis alignment on each video media element, and synthesizing into a finished video. According to the embodiment of the invention, full-process automatic processing from script generation to video synthesis can be realized, and a high-quality finished video meeting user requirements can be efficiently output. The method can be widely applied to the technical field of automatic editing.
Owner:GUANGZHOU FAISCO INFORMATON TECH

Weak supervision time sequence action positioning method and device based on semantic and significance knowledge cooperative propagation

The invention discloses a weak supervision time sequence action positioning method and device based on semantic and saliency knowledge cooperative propagation. The method comprises the following steps: 1) extracting time sequence features and saliency foreground features from an unclipped video; 2) constructing a basic branch and a saliency perception branch to respectively process the time sequence features and the saliency target features to obtain a basic class activation sequence, a motion and appearance representation score and a saliency class activation sequence, and performing weighted fusion on the four sequences to obtain a fusion action score sequence; 3) constraining interactive semantic information and significance information by using the consistency of branch distillation and branch action, and perfecting a fusion action score sequence; and 4) extracting key segments and ambiguous segments of the basic branches and the saliency perception branches, performing comparative learning by using the key segments and ambiguous segments between the branches and in the branches to improve feature representation, perfecting a fusion action score sequence in combination with a distillation result, and obtaining an action positioning result. According to the method, tiny human body actions and accurate time action boundaries can be perceived in an unedited video.
Owner:ZHEJIANG UNIV OF TECH