469 results about "Subtitle" patented technology

The invention discloses a multi-agent-based automatic generation method, system and terminal for a micro-drama, and the method comprises the steps: obtaining a user creation theme, and carrying out the story line generation processing and the split processing to obtain a plurality of split scripts which comprise commentaries, visual information, shot information and atmosphere information; performing dubbing conversion processing and subtitle generation processing according to the commentary to obtain a target dubbing audio and a target video subtitle; performing key frame generation processing according to the visual information to obtain a target key frame image, and performing video generation processing with the shot information to obtain a target video clip; performing background music generation processing according to the atmosphere information to obtain target background music; and performing time axis alignment processing and splicing processing on the target dubbing audio, the target video subtitles, the target video clips and the target background music to obtain a target synthesized episode. According to the invention, the production efficiency and the output quality of the micro-movie can be effectively improved.

The invention provides a multi-language cross-culture communication assisting method and system based on a large model. The method comprises the following steps: receiving a source language audio stream during a call, calling a multi-language sound frequency harmonic modulation feature library to extract fundamental frequency harmonic intensity distribution and tone turning features, and generating a cultural acoustic fingerprint vector; based on the vector, controlling a microphone array phase difference, directionally enhancing a fundamental frequency harmonic component of a speaker and suppressing noise, and outputting a high signal-to-noise ratio spectrogram; analyzing the pronunciation rhythm and tone turning characteristics of the spectrogram, capturing the pitch jump and duration of the syllable boundary, and generating an acoustic culture label; associating the spectrogram with a target semantic library, matching harmonic distribution and a cultural context rule based on a large model, and outputting a cultural interpretation prompt containing an ambiguity resolution suggestion; and generating a calibration result according to the acoustic tag and the semantic prompt, and overlapping the dynamic floating subtitles to the face area of the speaker in the video conference picture. According to the invention, cultural tone ambiguity in multi-language communication is eliminated.

The invention relates to the technical field of subtitle translation, in particular to a multi-modal time sequence alignment AI video translation method and system, and the method comprises the steps: 1, carrying out the multi-modal analysis of a to-be-translated video, and obtaining audio separation data, voiceprint feature data and visual time sequence data; 2, performing cross-language translation and context optimization on the basis of the voice of the audio separation data to generate a target language text, and synthesizing target language voice retaining the original voice color in combination with the voiceprint feature data and the target language text; generating a mouth shape animation matched with the target language voice based on the lip key point data and the limb action time sequence data; and step 3, performing four-dimensional alignment on the target language voice, the translated text, the mouth shape animation and the limb action sequence through a cross-modal time sequence encoder, and dynamically adjusting the layout of the bilingual subtitles to adapt to a video picture. According to the method and the device, multi-mode synchronization can be taken into consideration during video translation, so that the body actions such as voice, subtitles and mouth shapes are kept aligned.

The invention belongs to the technical field of speech translation, and particularly relates to a video speech automatic translation method based on a pre-training model, and the method comprises the following steps: 1, preprocessing video and audio data; step 2, voice recognition and language detection; step 3, machine translation and text post-processing; step 4, speech synthesis and audio mixing; step 5, synchronizing video processing and subtitles; step 6, quality control and multi-dimensional evaluation; 7, carrying out model iteration and data closed loop; and step 8, system deployment and engineering implementation. Through deep fusion of efficient transfer learning of the pre-training model and the multi-modal technology, a high-precision, low-cost and easy-to-expand video speech translation solution is constructed, the time and labor cost of globalized content production is greatly reduced, the cross-language communication efficiency is improved, immersive multi-language experience is provided, and the method is suitable for popularization and application. And a data-driven continuous optimization mechanism is established, so that the system performance is improved along with the increase of the use scale.

A system, a method and a processor are for AI-powered generation and delivery of video clips. The processor is configured to: load a first video file of a first video content item, the first video file comprising video frames associated with timestamps; load a first subtitle file of the first video content item, the first subtitle file comprising subtitle text associated with the timestamps; execute a natural language processing (NLP) model with the subtitle text as input, the NLP model including language pre-processing steps for classifying words, names or phrases in the subtitle text and associating initial classifiers with the subtitle text, the NLP model including one or more of a recurrent neural network (RNN), a Bidirectional Encoder Representations from Transformers (BERT) model, or a generative pre-trained transformer (GPT) model for a dialogue analysis comprising processing sequences of dialogue in the subtitle text in view of the initial classifiers to associate one or more portions of the dialogue with one or more first classifiers of first narrative elements; execute an image recognition model with at least some of the video frames as input, the image recognition model including a convolutional neural network (CNN) for an object detection analysis and a facial recognition analysis comprising processing video sequences to associate one or more of the video frames with one or more second classifiers of second narrative elements; generate a narrative map of the first video content item by temporally aligning the first narrative elements with the second narrative elements based on the timestamps associated with the video frames and the first subtitle file; and generate a video clip including at least one segment of the first video content item, the at least one segment including selected video frames associated with at least one of the first or second narrative elements identified from the narrative map and selected for inclusion in the video clip.

Automatic captioning pipelines and methods for automatically annotating video data with subtitles, which can be obtained using automatic speech recognition (ASR). An automatic captioning pipeline with inputs of multimodal data scales up the dataset of high-quality video-caption pairs. The automatic captioning pipeline generates video-caption pairs by establishing and using a large video-language dataset along with an automatic captioning approach leveraging multimodal inputs, such as textual video description, subtitles, and individual video frames.

The invention relates to a speech synthesis method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a source language dubbing audio corresponding to a source language subtitle text of a source video and a target language subtitle text translated by the source language subtitle text; extracting audio emotion features from the source language dubbing audio by using an emotion extractor, wherein the audio emotion features represent emotions expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence by using a text encoder to obtain text encoding features; fusing the audio emotion features with the text coding features to obtain emotion text features; and generating a target language audio based on the emotion text features by using a decoder, the target language audio being used as a dubbing audio of the source video in the target language. Therefore, the high-quality target language audio with the source dubbing audio emotion can be automatically and efficiently generated, the cost is low, and the efficiency is high.

The invention discloses an AI-driven video content subtitle synchronous translation method and system, and relates to the technical field of video subtitle synchronization, the system is combined with a video frame acquisition module and a face and lip action recognition module, and the system can accurately obtain the lip opening and closing vertical distance and opening and closing times of each role. The data are used for calculating the actual speaking speed, and the actual speaking speed is compared with a traditional speed index to obtain a first calibration difference coefficient. According to the method, the timestamps of the subtitles are effectively adjusted, the time deviation caused by the speech speed difference is reduced, the subtitles and the actual speech are more synchronous, and therefore the accuracy of the subtitles and the film watching experience of audiences are improved. The multi-person talking overlapping recognition module can accurately detect and mark the voice overlapping condition. And if the overlapped voice influence factor D exceeds the abnormal threshold F, the system triggers the second correction instruction to further calibrate the subtitle timestamp, so that the synchronization problem caused by voice overlapping is avoided.

The invention provides a video dubbing method and device, electronic equipment and a storage medium, and belongs to the technical field of video processing, and the method comprises the steps: separating a first audio in a to-be-dubbed video, and carrying out the sentence-by-sentence text conversion, and obtaining a first subtitle text with a time code; translating the first subtitle text into a second subtitle text with the same time code; normalizing the second subtitle text to obtain a third subtitle text according to the difference between the dubbing duration of the second subtitle text and the subtitle display duration; generating a third audio corresponding to the third subtitle text; and finally, synthesizing all the third audios, the background audio of the video to be dubbed and the silent video to obtain a final target video. According to the invention, under the condition that the audio duration and the subtitle display duration are different in the video dubbing process, the subtitle text is normalized, so that the damage to the original video file is avoided, and the high quality of video dubbing is ensured.

The invention discloses a character loss compensation processing device and method for video subtitle extraction, which effectively solve the problem of character loss caused by video image quality, complex background interference, OCR (Optical Character Recognition) limitation, dynamic subtitle change and the like. According to the method, key technologies such as self-adaptive subtitle denoising, semantic compensation and multi-frame fusion are adopted, and the completeness and accuracy of subtitle extraction are remarkably improved. The scheme is suitable for various types of movies, television dramas, short videos, conference videos and the like. The core innovation of the method is that lost or wrong characters can be effectively detected, compensated and corrected by intelligently analyzing the context and integrating multi-frame information, so that the subtitle output quality is greatly improved, and the effect is remarkable particularly in a scene that a single character is easy to lose.

The invention discloses a multi-mode-based short episode video editing method, system and device and a medium, and the method comprises the steps: carrying out the preprocessing of an original short episode video, and obtaining a second short episode video, subtitles and a subtitle timestamp; analyzing the subtitles, and generating a short episode abstract according to an analysis result of the subtitles; according to the short episode abstract and the highlight editing prompt, performing plot analysis and editing on the second short episode video, and performing editing to form a first video set; scoring videos in the first video set according to a multi-dimensional scoring rule, and screening out M highlight video clips before scoring; and adjusting the timestamps of the M highlight video clips according to the subtitle timestamps, and outputting the adjusted M highlight video clips. According to the method and the device, the full link from short video input to short play mixed video output is constructed, a user only needs to input the to-be-edited short play video, the mixed video can be automatically generated, and the video editing efficiency is improved while the video editing threshold is reduced.

The invention discloses a video editing method and device, electronic equipment and a storage medium. The method comprises the following steps: segmenting a to-be-edited video to obtain a plurality of video clips; understanding text generation and subtitle generation are carried out on each video clip, and a clip understanding text and a clip subtitle of each video clip are obtained; determining a target description text based on fragment understanding texts and / or fragment subtitles corresponding to a plurality of target video fragments selected from the plurality of video fragments; and performing video synthesis based on the plurality of target video clips and the target description text to obtain a target video corresponding to the to-be-edited video. According to the method provided by the invention, the video editing efficiency is relatively high, and the video editing cost is relatively low.

The invention belongs to the technical field of multimedia, and particularly relates to a short drama translation and explanation method based on a multi-mode neural network, which realizes full-process automatic processing from subtitle extraction to culture adaptation translation by integrating advanced technical means such as an OCR (optical character recognition) model, optical flow field analysis, an LSTM (long short term memory) network and a graph neural network. Particularly, the subtitles in a complex background can be accurately recognized and processed, subtitle traces are effectively removed, plot conflicts are understood through a deep learning algorithm, and then translation content conforming to target culture is generated. The method not only improves the accuracy of subtitle extraction, but also enhances the cultural adaptability of the translated content, so that the multimedia content can be more naturally and smoothly understood and accepted under different cultural backgrounds, and the watching experience of global users is greatly improved.

The invention discloses a method for intelligently extracting copyright information based on irregular dynamic subtitles. The method comprises the following steps: S1, inputting a program video sequence; s2, carrying out OCR (Optical Character Recognition) on the input program video sequence; s3, extracting copyright roles and entities in the video sequence; and S4, constructing a dynamic subtitle relation model, and generating a copyright information table. According to the method, technical means such as video text recognition, role extraction, role automatic verification and completion, entity extraction, entity knowledge enhancement verification and relation model matching based on dynamic subtitle multi-frame image fusion are adopted; according to the invention, accurate and automatic extraction of copyright information of directors, scripts, plans, words, compositions, singing and the like in program videos containing irregular subtitles is realized. According to the method, manual intervention is not needed, the copyright information extraction efficiency and accuracy are improved, and powerful support is provided for legal right protection of a video content creator.

The invention provides a video content textualization method based on multi-modal fusion, which comprises the following steps of: 1, dynamically identifying effective modal information existing in a video, including subtitle information detection, audio information detection and key frame information sampling; step 2, carrying out subtitle extraction on the subtitle information by adopting an OCR (Optical Character Region) enhancement method based on regional clustering; generating a voice text for the audio information by adopting a multi-engine collaborative transcription and weight fusion strategy; generating a descriptive text for the key frame information; step 3, performing space-time alignment and semantic fusion on the subtitles, the voice text, the descriptive text and the video time axis; and feeding back and adjusting a strategy of key frame information sampling according to a fusion result, and forming self-adaptive feedback. According to the textualization method, multi-modal information can be adaptively fused, and video total elements are covered, so that the robustness and comprehensiveness of content extraction are improved.

The invention provides a video processing method and device, equipment and a storage medium, and the method comprises the steps: segmenting a video material into a plurality of semantically coherent video clips, automatically extracting related video clips from the video material according to the text content provided by a user, and generating corresponding subtitles and voice broadcast ports matched with the related video clips, and finally synthesizing a section of finished video with clear logic and coherent content. According to the scheme, full-process automation from video material generation to final video file generation is realized, a large number of video materials can be quickly processed, the tedious manual editing process in traditional video production is reduced, the dependence on manual intervention is greatly reduced, the operation cost is reduced, the video production process is simplified, the video production efficiency is improved, and the user experience is improved. And the quality and the efficiency of finished products are also improved.

The invention provides a multi-mode content review method and device, a medium and a computer program, and relates to the technical field of content review. The multi-modal content review method comprises the following steps: calling a special recognition model to recognize subtitles and voice in target review content to obtain a character recognition result and a voice recognition result; the special identification model is a model obtained based on learning of a content sample and a subtitle label and a voice label of the content sample; and calling an artificial intelligence large model, performing fusion analysis and reasoning on the character recognition result, the voice recognition result and the video picture of the target review content according to the violation review rule, and outputting a violation behavior recognition result. The universality of the content review method is improved based on the good cognitive ability of the artificial intelligence large model; the artificial intelligence large model and the special small model are organically combined, the understanding cognitive ability of the artificial intelligence large model and the perception ability of the special small model are fully exerted, and accurate and effective examination of the content is realized.

The invention provides an explanation video generation method, device and equipment based on a large model and a medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of multimodality, natural language processing, computer vision, deep learning and the like. The method comprises the following steps: acquiring a plurality of subtitle texts and corresponding first timestamps in a to-be-processed video; based on the first timestamps of the plurality of subtitle texts, determining at least one subtitle-free fragment in the to-be-processed video and a corresponding second timestamp; performing visual content understanding on the at least one subtitle-free fragment by using a first multi-mode large model to obtain at least one subtitle complemented text corresponding to the at least one subtitle-free fragment; generating commentaries for the to-be-processed video by using a large language model based on the plurality of subtitle texts and the corresponding first timestamps and the at least one subtitle complemented text and the corresponding second timestamps; and generating a commentary video based on the commentary.

The invention provides a voice, picture and subtitle alignment method and system for real-time voice translation synthesis, and relates to the technical field of voice processing. According to the method, millisecond alignment of translated voice and video is realized through fragment-level parallel processing, compared with a traditional scheme, the delay is greatly reduced, and the problem that translated voice pictures are asynchronous during live broadcast is solved; the method comprises the following steps: automatically generating real-time voice translation according to live broadcast content, and respectively processing to obtain a corresponding srt file and a translated m3u8 fragment; therefore, there is no need to manually translate and generate subtitles in advance, and there is no need to replace the audio of the original video in advance, thereby saving the manpower cost, and providing a more universal live broadcast stream subtitle scheme. Besides, when the languages are switched, the player can automatically select the starting time point, so that the method supports multiple playing modes such as online live broadcast and live broadcast playback.

To solve the problem of saving time and labor in a system for automatically cutting out and re-editing broadcast highlights, and making it easy to create attractive highlight videos.SOLUTION: A system for automatically cutting out and re-editing broadcast highlights uses AI to automatically extract exciting scenes, generate subtitles, and add sound effects.SELECTED DRAWING: Figure 1

The invention relates to the technical field of computers, and discloses a manuscript generation method, system and device and a storage medium, and the manuscript generation method comprises the steps: carrying out the audio recognition of a video file to obtain an audio recognition result, and carrying out the recognition of subtitles of the video file to obtain a subtitle recognition result; adding a segmentation punctuation in the audio recognition result, and obtaining audio sub-strings segmented according to a target segmentation punctuation; for any audio sub-character string, according to the timestamp of the audio sub-character string, acquiring a subtitle sub-character string located in the same time period as the audio sub-character string from the subtitle recognition result, and correcting the audio sub-character string according to the subtitle sub-character string; and outputting the corrected audio recognition result as a video manuscript of the video file.

The embodiment of the invention provides a video processing method and device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of multimedia. The video processing method comprises the following steps: playing a target video; wherein the target video comprises a current video frame embedded with a target subtitle; under the condition that a first preset condition is triggered, displaying a modifiable identifier for the target subtitle on the current video frame; and popping up an editing entry for modifying the target subtitle in response to the triggering of the region where the modifiable identifier is located, receiving current editing content through the editing entry; and uploading the current edited content to a server, so that the server replaces the target subtitle with the current edited content. According to the technical scheme provided by the embodiment of the invention, the published video can be quickly corrected. And the man-machine interaction efficiency is improved through a convenient mode of modifying the identifier and editing the entry.

A system and method for programmatic media preview generation, including: a preview generation system executing on a computer processor and configured to receive a request to generate a preview video of a source video file; select a source video for analysis; obtain a set of text metadata comprising groupings of subtitles of the source video; invoke a machine learning model using the set of text metadata to infer a set of candidate previews for the source video file; and provide a final set of candidate previews in response to the request; and a ranking module comprising functionality to rank the set of candidate previews to generate the final set of candidate previews.

The invention provides a system and a method for automatically aligning time axis dubbing, and the system comprises a silent video extraction module which is used for removing the audio of an original video; the OCR subtitle extraction module is used for generating a subtitle file in an SRT format; the AI translation module translates the subtitles into a target language; the subtitle erasing module is used for positioning and erasing original subtitles in the video; the TTS speech synthesis module is used for generating a dubbing file; the time axis alignment module is used for dynamically adjusting the speech speed, the video speed and the subtitle time; and the video synthesis module is used for integrating the silent video, dubbing and subtitles. According to the method, OCR subtitle extraction, AI subtitle translation and generative dubbing technologies are combined, and the voice speed of the voice is dynamically adjusted to keep the voice aligned with the time axis of the video and the time axis of the subtitle, so that the quality of the translation play can be improved, the cost can be greatly reduced, the product marketing is accelerated, and great convenience and competitive advantages are brought to content creators and issuers.

The invention provides a short play subtitle translation system based on artificial intelligence, and relates to the technical field of artificial intelligence, and the system comprises a subtitle recognition module, an erasing module, a translation module and an output module, and can automatically recognize time information, position coordinates and visual style parameters of subtitles in a short play video. And generating a picture sequence without original subtitles in combination with the erasing processing, and translating the recognized original subtitle text into target language subtitles. The system generates target rendering parameters based on explicit style parameters and implicit style embedding vectors, realizes high restoration of fonts, strokes, shadows, gradient, transparency, textures and dynamic special effects, and performs adaptive adjustment according to target subtitle text features. Therefore, the visual consistency and culture adaptability of translated subtitles are improved in a short drama scene with complicated subtitle styles, frequent dynamic changes and obvious cross-culture differences, the audience impression is improved, and the manual post-processing workload is reduced.

The invention provides a foreign language teaching video generation method and device and a computer program product. The foreign language teaching video generation method comprises the steps of obtaining a video subtitle text corresponding to a to-be-processed video; based on the video subtitle text and a preset explanation generation rule, a foreign language teaching explanation text of the video subtitle text is generated by adopting a large language model; generating a corresponding explanation audio based on the foreign language teaching explanation text; determining a time information relationship between the foreign language teaching explanation text and the explanation audio; according to the time information relationship between the foreign language teaching explanation text and the explanation audio, generating a display style corresponding to the explanation audio; and generating an explanation video corresponding to the to-be-processed video according to the explanation audio, the display style corresponding to the explanation audio and the time information relationship between the explanation text and the explanation audio. According to the invention, the target teaching video corresponding to the to-be-processed video can be automatically generated, the generated teaching video is highly targeted, and personalized requirements of users can be met.

A system, a method and a processor are for AI-powered generation and delivery of video clips. The processor is configured to: load a first video file of a first video content item, the first video file comprising video frames associated with timestamps; load a first subtitle file of the first video content item, the first subtitle file comprising subtitle text associated with the timestamps; execute a natural language processing (NLP) model with the subtitle text as input, the NLP model including language pre-processing steps for classifying words, names or phrases in the subtitle text and associating initial classifiers with the subtitle text, the NLP model including one or more of a recurrent neural network (RNN), a Bidirectional Encoder Representations from Transformers (BERT) model, or a generative pre-trained transformer (GPT) model for a dialogue analysis comprising processing sequences of dialogue in the subtitle text in view of the initial classifiers to associate one or more portions of the dialogue with one or more first classifiers of first narrative elements; execute an image recognition model with at least some of the video frames as input, the image recognition model including a convolutional neural network (CNN) for an object detection analysis and a facial recognition analysis comprising processing video sequences to associate one or more of the video frames with one or more second classifiers of second narrative elements; generate a narrative map of the first video content item by temporally aligning the first narrative elements with the second narrative elements based on the timestamps associated with the video frames and the first subtitle file; and generate a video clip including at least one segment of the first video content item, the at least one segment including selected video frames associated with at least one of the first or second narrative elements identified from the narrative map and selected for inclusion in the video clip.

The embodiment of the invention discloses a subtitle processing method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a first subtitle file; extracting the time axis in each piece of subtitle information to obtain subtitle content in each piece of subtitle information; obtaining translation prompt information of the first subtitle file, and translating each subtitle content under the prompt of the translation prompt information to obtain the translated content of each subtitle content; and performing reconstruction processing on the translation content of each subtitle content and the time axis corresponding to each subtitle content to generate a second subtitle file. By adopting the embodiment of the invention, the translation accuracy and translation efficiency of subtitle translation can be improved.

The invention discloses a virtual digital human multimedia teaching interaction method and system and a storage medium, and the method comprises the steps: calculating an audio mouth shape time difference, a mouth shape subtitle time difference, an audio subtitle time difference and a video mouth shape time difference based on a same-window data group on the basis of a unified time aperture and a window, comparing the time differences with a time difference threshold value, and outputting a dislocation alarm result, unified constraint on four types of key alignment relationships is realized; when an alarm is triggered, selecting a degradation strategy according to a dislocation alarm result and generating an execution record, and summarizing the dislocation alarm result and the execution record to form an evidence chain data packet; when the cumulative number of times in the preset window number exceeds a diffusion blocking threshold value, recording output is switched into a video stream which retains audio and subtitles and does not overlap mouth shapes; and generating a correction release stream based on the evidence chain data packet, and replacing the recorded broadcast corresponding time slice and retaining the Hash verification value when the time sequence consistency index meets the release threshold, thereby realizing drivable treatment action, suppressible diffusion and replaceable correction.