6859 results about "Video based" patented technology

A video localization system localizes actions in videos based on a classification model and an actionness model. The classification model is trained to make predictions of which segments of a video depict an action and to classify the actions in the segments. The actionness model predicts whether any action is occurring in each segment, rather than predicting a particular type of action. This reduces the likelihood that the video localization system over-relies on contextual information in localizing actions in video. Furthermore, the classification model and the actionness model are trained based on weakly-labeled data, thereby reducing the cost and time required to generate training data for the video localization system.

A video localization system localizes actions in videos based on a classification model and an actionness model. The classification model is trained to make predictions of which segments of a video depict an action and to classify the actions in the segments. The actionness model predicts whether any action is occurring in each segment, rather than predicting a particular type of action. This reduces the likelihood that the video localization system over-relies on contextual information in localizing actions in video. Furthermore, the classification model and the actionness model are trained based on weakly-labeled data, thereby reducing the cost and time required to generate training data for the video localization system.

The embodiment of the invention provides a three-dimensional model adaptive generation method based on video and point cloud data, scene data is collected through a multi-view camera array and a depth scanning array, and key features are extracted and screened by using a hierarchical feature extraction network and an attention mechanism. And organizing local data blocks by adopting a hierarchical index tree based on a union-check set, and realizing self-adaptive parameter optimization by combining integrity and accuracy scores. Meanwhile, efficient reconstruction and material mapping of a local three-dimensional grid are realized through a parallel operation channel and a dynamic calculation resource scheduling mechanism. According to the method, the precision and efficiency problems of a traditional three-dimensional reconstruction scheme in processing a large-scale scene are effectively solved, and the model generation quality is improved.

The invention discloses a video analysis-based electric power operation personnel behavior identification and early warning system and method, which realize accurate identification and real-time early warning of electric power operation personnel behaviors by combining a video analysis technology with multi-modal data fusion, and effectively improve the safety management level of an operation site. Compared with a traditional safety supervision mode, the method employs a mode of combining deep learning target detection and time sequence behavior analysis, improves the recognition accuracy of operators and safety equipment, fuses the data of equipment worn by the operators with video data, improves the detection precision, and improves the safety supervision accuracy. And misjudgment caused by illumination change, shielding or complex environment is reduced. Besides, high-risk violation behaviors such as no safety helmet wearing, no safety belt fastening, violation climbing, high-altitude object throwing and the like are accurately recognized through the violation detection module, and different levels of alarm measures are adopted according to the severity of the violation behaviors in combination with an early warning feedback mechanism, so that the pertinence and response efficiency of early warning are improved.

Disclosed is an area monitoring system including a camera module having configured to generate an output representative of a captured image and / or a video of an area including a object. The system includes a state machine including an inference engine configured to: identify and track an object; establish a state and / or a change in state of an object via machine vision inferencing; and generate an output representative of an object identification, an object state, and / or a change in an object's state. The state machine module includes a mapping engine configured to: process the camera module output to assign coordinates to an object contained in an image and / or a video based on a coordinate system of a computer vision map of an area; apply a timestamp in connection with assigning coordinates an object; and generate a coordinate timeseries log for plural coordinate-timestamp pairs.

The invention provides a tunnel construction monitoring analysis method and system based on AI video monitoring, and relates to the technical field of artificial intelligence. A tunnel construction original video stream data set is acquired through a multi-view video acquisition device, and a spatio-temporal feature sequence is generated through spatio-temporal feature fusion processing; the spatiotemporal feature unit comprises a pixel region movement track, a personnel wearing identifier and a construction equipment contour feature, then extracting dynamic features of the spatiotemporal feature sequence to obtain a dynamic spatiotemporal feature set, and executing construction personnel statistical processing based on the dynamic spatiotemporal feature set; generating a hole entering personnel quantity statistical result, a personnel distribution statistical result of each construction part and a personnel identity matching result, performing work efficiency comparative analysis according to the personnel statistical result and a preset construction progress plan, and generating a construction progress deviation index and resource allocation optimization strategy; and when the progress deviation index exceeds the first threshold value, early warning is triggered, and a strategy is output to the management terminal, so that effective monitoring and management of tunnel construction are realized.

The invention discloses an automatic labeling and acquisition standardization method and system based on video content, and the method comprises the steps: decoding a video file into a processable video frame sequence, carrying out the standardization processing, unifying the format and the coding standard, and adjusting the resolution and the frame rate; performing content analysis on the standardized video frame sequence, extracting key frames, identifying objects, scenes and actions, and generating a video analysis result and a preliminary label; according to a video content analysis result, separating an audio stream from the video to carry out noise reduction processing and voice-to-text analysis, and combining the processed text with the video content analysis result to generate multi-modal comprehensive annotation information; based on the comprehensive labeling information, taking feedback of the user in video retrieval, label clicking and watching behaviors as an adjustment basis of label weights, and updating weight distribution of labels by analyzing user behavior data; labeling information and weight distribution are integrated into an index database, and indexes are updated in real time according to user behaviors and content changes.

Systems and methods for generating and providing a virtual walkthrough interface can include generating a virtual walkthrough video based on view synthesis renderings generated by neural radiance field model. The neural radiance field model can be trained based on a plurality of images of an environment and may generate the view synthesis renderings based on processing positions along a determined walkthrough path. The generated virtual walkthrough video can then be scrubbed through to provide the virtual walkthrough interface.

The invention discloses a video feature extraction and multi-dimensional matching-based movie advertisement real-time pushing system, and belongs to the technical field of digital advertisements. According to the system, multi-modal analysis is carried out on visual, audio, text and semantic features of a movie through a video feature extraction module, a dynamic interest tag constructed by a user portrait analysis module is combined, and the weighted matching degree of a video scene, user preference and an advertisement tag is calculated by utilizing a multi-dimensional matching engine; and millisecond-level advertisement putting is completed through the real-time pushing decision module. According to the method, context awareness and streaming computing technologies are fused, the relevance between advertisements and content scenes is remarkably improved, cross-platform deployment is supported, the method is suitable for short videos, live broadcast and other real-time scenes, the advertisement click rate is increased by 55%-72% through tests, the response delay is lower than 200 ms, and the method has the technical advantages of being efficient, accurate and low in delay.

Various disclosed embodiments are directed to learning-free video object segmentation (VOS) of one or more video frames. In various instances, such VOS does not rely on human supervision or labeling and can be generalized or applied to any video “in the wild.” This is because training and fine-tuning are not required for VOS. Particular embodiments perform VOS based on feature similarity of different sections of a video frame and / or estimated motion similarity of different sections of the video frame, such as via a graph cut on a graph.

The invention relates to the technical field of monitoring analysis, in particular to a park safety monitoring method and system based on dynamic visual identification. The method comprises the following steps: acquiring a plurality of park monitoring videos based on park monitoring equipment; performing three-dimensional illumination migration rendering on the plurality of park monitoring videos, and performing low-frame smooth completion so as to generate a time sequence monitoring image frame sequence; carrying out spatial position positioning calculation on the park monitoring equipment, carrying out topological association network reconstruction, and constructing an omnibearing monitoring perception topological map; performing dynamic object visual detection on the time sequence monitoring image frame sequence, and performing frame-by-frame optical flow tracking analysis to generate a dynamic target optical flow trajectory; and carrying out multi-time-point sampling on the dynamic target optical flow trajectory, and then carrying out mutation point identification so as to generate an optical flow trajectory mutation point. According to the invention, efficient park safety monitoring analysis and high-precision safety early warning are realized.

A multimodal search system using a video query is described. The system can receive video data captured by a camera of a user device. The video data can have a sequence of image frames. Additionally, the system can receive audio data associated with the video data captured by the user device. Moreover, the system can process, using one or more machine-learned models, the sequence of image frames to generate video embeddings related to the sequence of the image frames. The video embeddings can have a plurality of image embeddings associated with the sequence of image frames. Furthermore, the system can determine one or more video results based on the video embeddings and the audio data. Subsequently, the system can transmit, to the user device, the one or more video results.

Systems and methods for video summary generation are disclosed. A chat and video conference provider generates a text summary for a meeting video and an audio summary based on the text summary. The chat and video conference provider determines a first set of correspondences between a first set of video portions from the meeting video and portions of the text summary based on a transcript of the meeting video. The chat and video conference provider determines a second set of correspondences between a second set of video portions from the meeting video and the portions of the text summary based on image data of the meeting video. The chat and video conference provider selects a plurality of video frames from the first set of the correspondences and the second set of correspondences to generate a video summary of the meeting video based on the audio summary.

The invention discloses an intelligent power plant joint inspection system and method based on video AI and a robot, and the method comprises the steps: dividing an intelligent power plant into a plurality of subareas, evaluating the danger levels of the plurality of subareas according to historical data, and obtaining the danger levels of the plurality of subareas; setting an inspection cycle according to the danger levels of the plurality of partitions, and setting a plurality of robots to acquire operation data according to the inspection cycle to obtain inspection data of the plurality of partitions; acquiring a plurality of monitored videos, and performing video analysis on the acquired monitored videos by using an AI technology to obtain an abnormal position; according to the abnormal position, distributing an inspection robot of a corresponding partition to carry out fixed-point inspection to obtain fixed-point inspection data; and performing fault detection on the periodic inspection data and the fixed-point inspection data by using the fault recognition model, and performing fault early warning according to a fault detection result. The invention relates to the technical field of power plant inspection, and solves the technical problems of unreasonable inspection resource distribution and untimely fault early warning in an existing combined inspection system.

A method implemented by a video coding apparatus. The method includes applying a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample. The NN filter includes an NN filter model generated based on partitioning information of the video unit. Usage of the NN filter is indicated by one or more syntax elements in a bitstream. A conversion is performed between a video media file and a bitstream based on the filtered sample.

A computer-implemented method comprising: receiving, from a user device, video data from a user; training a first machine learning model based on the video data to provide a second machine learning model, the second machine learning model being personalized to the user, wherein the second machine learning model is trained to predict movement of the user based on audio data; receiving further audio data from the user; determining predicted movements of the user based on the further audio data and the second machine learning model; using the predicted movements of the user to generate animation of an avatar of the user.

The invention discloses an underground mine operation state analysis system and method based on video monitoring data, and the system comprises a data collection module which is used for collecting mine video and environment parameter data through a distributed sensor network, and generating a multi-dimensional data fusion set based on a space-time label technology; the edge analysis module is used for extracting feature parameters through a convolutional neural network algorithm based on the multi-dimensional data fusion set and generating a mine operation state recognition result; the fence construction module is used for constructing a three-dimensional digital model and a dynamic safety boundary based on the mine operation state recognition result to form a real-time monitoring reference framework; and the decision execution module is used for performing hierarchical risk assessment on the monitoring data in the security boundary based on the real-time monitoring reference framework, and generating a security early warning and disposal scheme with a tracing identifier. Each piece of early warning and disposal information is attached with a unique tracing identification code, so that follow-up event backtracking analysis is facilitated, and the risk management and control capability is continuously improved.

An action localization method, device, electronic equipment, and computer-readable storage medium are provided. The action localization method includes: identifying at least one target video segment containing a target object in a video; acquiring a first action recognition result of at least one image frame in the at least one target video segment and a second action recognition result of the target video segment; and acquiring an action localization result of the video based on the first action recognition result and the second action recognition result.

The invention belongs to the technical field of artificial intelligence, and particularly relates to a video analysis processing system and method based on a multi-modal large model, and the method comprises the steps: enabling the multi-modal control large model to receive a task which is input by a user and needs to be completed through natural language description, carrying out the semantic analysis of the input of the user, extracting task information, and carrying out the analysis of the task information; a task related to video understanding is decomposed into sub-tasks and the sub-tasks are sent to a video understanding module; the video understanding module receives a video or a video clip required by a task, executes a subtask to analyze the video, and generates embedding and description of video content to be input into a multi-modal control large model; when the multi-modal control large model receives a query request of a user, a query task of the user is analyzed into executable subtasks and mapped to a video understanding task, and a response result based on the query request is generated based on embedding and description of video content generated by the video understanding large model. Therefore, effective fusion and mutual enhancement of cross-modal information are realized, and the video understanding precision is improved.

The invention discloses a long video role scene decoupling generation method and system based on multi-modal diffusion, and relates to the technical field of image processing, and the method comprises the steps: S1, synthesizing the advanced features of a role and a scene through a SigLIP encoder and a DINOv2 encoder; s2, performing cross-modal feature fusion on the advanced features to obtain joint features, and compressing the joint features to obtain compact vectors; s3, generating text features according to the text prompt; s4, potential codes are generated from an input video through a causal 3D convolution encoder, the potential codes pass through a linear projection matrix and then are spliced with a memory state for dimension reduction, and a segmented potential vector sequence is obtained; s5, performing decoupling perception generation on the segmented potential vector sequence through an improved 3D-UNet, and performing deconvolution up-sampling reconstruction after deterministic sampling to obtain an RGB video segmented sequence; according to the method, the key problems of rough dynamic control, limited generation length and over-high resource consumption in long video generation are solved, and the quality and efficiency of the generated video are remarkably improved.

An active sparse labeling system that provides high performance and low annotation costs by performing partial instance annotation (i.e., sparse labeling) by frame level selection to annotate the most informative frames, thereby improving action detection task efficiencies. The active sparse labeling system utilizes a frame level cost estimation to determine the utility of each frame in a video based on the frame's impact on action detection. The system includes an adaptive proximity-aware uncertainty model, which is an uncertainty-based frame scoring mechanism. The adaptive proximity-aware uncertainty model estimates a frame's utility using the uncertainty of detections of the frame's proximity to existing annotations, thereby determining a diverse set of frames in a video which are effective for learning the task of dense video understanding (such as action detection). In addition, the active sparse labeling system includes a loss formulation training model (max-Gaussian weighted loss) that uses weighted pseudo-labeling.

The invention discloses a video analysis-based multi-scene operator violation behavior identification method and system, and belongs to the technical field of intelligent operation safety monitoring and artificial intelligence identification, and the method comprises the steps: collecting a real-time video stream of a multi-scene operation site; recognizing a continuous action time sequence in the real-time video stream by using an action recognition depth model; constructing the continuous action time sequence into an action behavior sequence; the action behavior sequence is constructed into a directed behavior graph with time, space and action labels, the directed behavior graph is compared with a directed behavior graph corresponding to the standard action behavior sequence, and illegal behaviors are recognized; and carrying out multi-mode early warning on the identified illegal behaviors. According to the method, the bottleneck that the traditional image recognition technology is weak in action sequence semantic understanding and poor in environmental adaptability is broken through, and accurate recognition and real-time early warning of illegal behaviors in multi-scene operation are achieved.

The invention provides a real-time video translation and audio and picture synchronization method and system based on a multi-modal large model, and relates to the technical field of video translations, and the method comprises the steps: obtaining a source video; extracting the source video based on the multi-modal large model to obtain a multi-modal feature; fusing the multi-modal features through a cross-modal attention mechanism to generate a context semantic vector; translating into a target language text in real time based on the context semantic vector, and processing the translated language text based on the multi-modal features to obtain a translated language sound source; and performing mouth shape adjustment on the source video based on the translation language sound source, and merging the translation language sound source and the mouth shape animation video to obtain a real-time translation video with synchronous sound and picture. According to the method, the limitation of traditional single-modal translation is broken through, and the semantic accuracy of translation is remarkably improved by dynamically aligning the context information through the multi-modal features in combination with a cross-modal attention mechanism.

Systems, methods, and computer-readable media are disclosed for systems and methods multimodal indexing of video using machine learning. An example method may include deceiving, by a video encoder of an audio-video transformer neural network comprising one or more computer processors coupled to memory, a first frame and a second frame associated with a first segment of a video. The example method may also include receiving, by an audio encoder of the audio-video transformer neural network, an audio spectrogram comprising first audio data associated with the first segment of the video. generating, by the video encoder, a first video embedding. The example method may also include generating, by the audio encoder, a first audio embedding. The example method may also include determining a fusion of the first video embedding and the first audio embedding using a multimodal bottleneck token. The example method may also include determining an output including the first video embedding and the first audio embedding. The example method may also include determining a classification of the first portion of the video based on the output.

A client device may be enabled to render a three-dimensional [3D] scene comprising one or more objects. At a server system, a video-based representation of an object may be streamed as one or more video streams to the client device. At the client device, the scene may be rendered from a viewing position within the scene to obtain a rendered view of the scene. Furthermore, a relative position between the viewing position and a position of the object in the scene may be determined. At the server system, the one or more video streams may be generated to show the object from a limited set of viewing angles, wherein the limited set of viewing angles is selected based on the relative position. At the client device, a viewing angle may be selected from the limited set of viewing angles and the video-based representation of the object may be placed at said selected viewing angle in the scene.

The invention discloses a gesture action video generation method and device based on fine-grained semantic description, and the method comprises the steps: carrying out the preprocessing and data enhancement of an original video, and constructing sample data composed of a video frame and a fine-grained semantic description text used for describing a gesture action in the video frame; constructing a training architecture consisting of a video coding module, a text coding module, a noise reduction generation module and a video decoding module; a loss function is constructed based on the difference between added noise in the video coding module and de-noised noise in the de-noising generation module, only parameters of the de-noising generation module are optimized by using the loss function, and a video generation model is formed by the text coding module, the de-noising generation module and the video decoding module after the parameters are optimized; and generating a gesture action video based on the input random vector and the fine-grained semantic description text for describing the gesture action by utilizing a video generation model. In this way, a high-quality general digital human video containing gesture actions is generated based on the fine-grained semantic description text.

Devices, systems, and methods for encoding and decoding digital video using a lookup table (LUT) containing motion candidates are described. In a representative aspect, a method of video processing includes: maintaining one or more tables, wherein at least one of the one or more tables is associated with one or more entries; deriving a second set of motion candidates based on a first set of motion candidates in the at least one table; and performing a conversion between a current block of a video and a bitstream representation of the video based on the second set of motion candidates.

Methods are provided for generating distilled multimedia data sets tailored to user's persona and / or task(s) to be performed associated with an enterprise network and enable interactive contextual learning using a multi-modal knowledge graph. Methods involve obtaining multimedia data from one or more data sources related to operation or configuration of an enterprise network and determining context for generating a distilled multimedia data set based on at least one of user input and user persona. The methods further involve generating, based on the context, the distilled multimedia data set that includes a set of multimedia slices generated from the multimedia data using a multi-modal knowledge graph. The multi-modal knowledge graph is generated using a graph neural network and indicates relationships among a plurality of slices of the multimedia data. The methods further involve providing the distilled multimedia data set for performing one or more actions associated with the enterprise network.

The invention discloses an intelligent parking space state recognition system and method based on video detection, and belongs to the technical field of parking space state recognition, and the method specifically comprises the steps: collecting the video data of a parking area, carrying out the preprocessing of the collected video data of the parking area, carrying out the modeling of a parking lot background based on a time sequence, and carrying out the real-time updating, based on a background modeling result, a foreground is detected, a foreground area is extracted, contour analysis is carried out on the detected foreground area, the occupation state of each parking space is judged, a real-time parking space distribution diagram of the parking lot is generated according to the occupation states of all the parking spaces, and the occupation or idle state of each parking space is displayed. The intelligent parking space state recognition system based on video detection can keep high accuracy in actual scenes such as complex illumination conditions, complex environments, different vehicle types and angles and the like, helps a driver to quickly find an idle parking space, and reduces traffic jam caused by parking space finding.

The invention discloses an abnormal behavior security early warning system based on multi-video image analysis, and relates to the field of security early warning, and the system comprises a path construction module which constructs a security path based on the installation position of video collection equipment, and constructs an abnormal behavior comparison library; the feature comparison module is used for constructing a security comparison model by using an algorithm, identifying abnormal behaviors of personnel, performing quantitative analysis to obtain a comprehensive risk index, and determining abnormal security personnel; the personnel judgment module is used for grading the security paths and determining security risk personnel; the personnel analysis module is used for judging the risk judgment value change condition of security risk personnel and determining a priority troubleshooting path; the regional analysis module is used for determining a dangerous region and security and protection high-risk personnel through clustering analysis; and the model optimization module is used for checking high-risk suspicious personnel to obtain false marks and missing personnel, and optimizing a security comparison model. The security early warning capability can be improved, and the method is suitable for various security scenes.