Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

439 results about "Visual marking" patented technology

Method and system for designing a stent

There is described a computer-implemented method for designing a stent, comprising: providing for display a first image of a 3D model of at least a portion of a body part comprising an aneurysm; receiving a position for each one of a plurality of reference points chosen along the body part; inserting in the first image a visual marker at the position of each reference point, thereby obtaining a second image; providing the second image for display; for each reference point, calculating a respective diameter; calculating at least one length using the positions of the reference points; and outputting the respective diameter and at least one length.
Owner:VITAA MEDICAL SOLUTIONS INC

Automatic driving interpretation text determination method based on large visual language model

The invention relates to the technical field of automatic driving, and particularly discloses an automatic driving interpretation text determination method based on a large visual language model, and the method comprises the steps: obtaining a multi-view camera image, a BEV map image and a text query template; video features in the multi-view camera image are extracted, and image features in the BEV map image are extracted; aligning the image features with the video features based on a modal encoder; encoding the text query template, generating a text mark, mapping the aligned image features and video features to a text embedding space based on a shared projector, generating a visual mark, and inputting the visual mark and the text mark into an LLM backbone model; comprehensive three-dimensional dynamic / static situation perception and visual representation unification of a local scene and a global map are realized, the 3D LVLM architecture has excellent three-dimensional space understanding and dynamic time reasoning capabilities, and the accuracy and safety of end-to-end automatic driving are effectively improved.
Owner:JILIN UNIVERSITY

Controllable image-to-video generation

Techniques are generally described for controllable image-to-video generation. In various examples, a first image representing at least a first object may be received. First input data including a selection of the first object in the first image for animation may be received. Second input data including at least a first bounding box indicating a target location of the first object may be received. A latent diffusion text-to-image model and the first image may be used to generate a first plurality of visual tokens. One or more first grounding tokens may be generated representing a location of the first bounding box. The latent diffusion text-to-image model may be used to generate a video animating the first object based on the first plurality of visual tokens and the one or more first grounding tokens.
Owner:AMAZON TECH INC

Multi-modal knowledge graph completion model training method, completion method and device

The invention provides a multi-modal knowledge graph completion model training method, a completion method and equipment, and relates to the field of computer systems based on specific calculation models. The method comprises the following steps of: extracting an image block feature vector, a word feature vector, a structure feature vector, a corresponding visual mark, a text mark and a structure mark by adopting a multi-modal knowledge graph completion model; based on each visual mark, the text mark and the structure mark, acquiring multi-modal fusion feature data of each entity by a graph attention mechanism; and complementing the original multi-modal knowledge graph according to each piece of multi-modal fusion feature data, and calculating the target loss of the complemented multi-modal knowledge graph to optimize the multi-modal knowledge graph complementing model. According to the method, fine-grained multi-modal feature extraction and application can be realized in the completion model training process, the reliability and effectiveness of the completion model training process can be effectively improved, and then the accuracy and reliability of multi-modal knowledge graph completion can be improved.
Owner:BEIJING UNIV OF POSTS & TELECOMM

Memory enhanced vision-language-motion submerged space dynamic fusion automatic driving method

The invention relates to a memory enhanced vision-language-action submerged space dynamic fusion automatic driving method. Comprising the following steps: generating a bird's-eye view feature map; extracting scene, agent and map marks from the aerial view feature map, and fusing the mark at the current moment and the previous i historical marks to generate memory enhanced visual marks; the memory enhanced visual mark and the vehicle state information are converted into a submerged space, text input of a driver is marked and unified into the submerged space, the submerged space is represented and fused, and fusion representation is generated; an automatic driving instruction data set is introduced for adjustment, and a large language model adapting to an automatic driving task is obtained; according to the fusion representation, track planning is carried out in an autoregression mode by using a large language model, and path point coordinates are obtained; and designing a transverse controller and a longitudinal controller based on PID (Proportion Integration Differentiation) to track and control the coordinates of the path points. According to the invention, the visual representation capability of end-to-end driving is improved, and the visual-language-action fusion effect is improved.
Owner:NANJING UNIV OF SCI & TECH

Data processing method and system, electronic equipment, storage medium and computer program product

The invention discloses a data processing method and system, electronic equipment, a storage medium and a computer program product, and relates to the technical field of large model technology and key value caching. The method comprises the following steps: acquiring a plurality of visual marks, a plurality of text marks and initial key value data corresponding to a visual language model; determining a key text mark in the plurality of text marks by utilizing the initial key value data; according to the initial key value data and the distribution positions of the key text marks in the multiple text marks, importance evaluation is conducted on the multiple visual marks, an evaluation result is obtained, and the evaluation result is used for representing cross-modal attention weight distribution between the multiple visual marks and the key text marks; and according to an evaluation result, performing cache compression processing on the initial key value data to obtain target key value data. According to the method and the device, the technical problem that the model reasoning efficiency is influenced due to high key value data caching overhead of a visual language model in related technologies is solved.
Owner:ALIBABA CLOUD FEITIAN (HANGZHOU) CLOUD COMPUTING TECH CO LTD

In-frame video markers

Techniques are described for performing actions (e.g., streaming video content, such as advertising content) based on visual markers detected within a live video feed being streamed to one or more computing devices. First video content is streamed to the one or more computing devices. A presence of a content item within the first video content is detected at a first point in time during streaming of the first video content. An action associated with the content item is performed at a second point in time in response to detection of the presence of the content item in the first video content.
Owner:AMAZON TECH INC

Light rail vehicle safety monitoring system based on multi-sensor fusion

The invention discloses a light rail vehicle safety monitoring system based on multi-sensor fusion, and relates to the technical field of light rail vehicle safety monitoring, the system is composed of a plurality of functional modules, and the system comprises a multi-mode sensing module for capturing images of the surrounding environment of a light rail in real time and delimiting a visual mark area; performing three-dimensional scanning on the visual marking area, generating environment point cloud data in real time, and acquiring a dynamic object distance by using a Doppler compensation algorithm; the data processing module is used for calculating object deformation between adjacent frames by adopting an optical flow method and a CNN deformation detection network; carrying out object relative motion calculation according to object shape analysis; carrying out instance segmentation based on MaskR-CNN, and outputting an object contour; meanwhile, a dynamic target clustering algorithm is used for conducting multi-target tracking on the object contour, and three-dimensional coordinates and motion vectors of the object are output in combination with environment point cloud data; and the fusion and analysis module performs time synchronization based on the PTP protocol.
Owner:HANGZHOU JIAHE INTELLIGENT TECH CO LTD

A transformer-based reprsentation-learning model with unified processing of multimodal input for clinical diagnostics and prognostics

PCT designated stage expiredWO2024254721A1Image enhancementMedical data miningVisual markingDisease
During the diagnostic process, clinicians leverage multimodal information, such as the chief complaint, medical images, and laboratory-test results. Deep-learning models for aiding diagnosis have yet to meet this requirement. In some aspects, a transformer-based representation-learning model can be used as a clinical-diagnostic aid that processes multimodal input in a unified manner. Rather than learning modality-specific features, the model uses embedding layers to convert images and unstructured and structured text into visual tokens and text tokens, and bidirectional blocks with intramodal and intermodal attention to learn holistic representations of radiographs, the unstructured chief complaint and clinical history, structured clinical information such as laboratory-test results and patient demographic information. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary disease (by 12%and 9%, respectively) and in the prediction of adverse clinical outcomes in patients with COVID-19 (by 29%and 7%, respectively). Leveraging unified multimodal transformer-based models may help streamline the triaging of patients and facilitate the clinical decision process.
Owner:GAO YUANXU +1

Image-text processing method and device for marking compression framework

The invention discloses an image-text processing method and device for marking a compression framework. The method comprises the following steps of: extracting visual features; a visual mark screening processing step; a text feature extraction step; and a multi-modal fusion and model processing step. The method has the beneficial effects that the inference efficiency of the MLLMs is remarkably improved under the condition that the visual mark compression frame does not need additional training; through global and local information fusion of the DVTS module and text guide supplement of the TGVC module, the number of visual marks is greatly reduced, meanwhile, key visual information is reserved, and visual-text alignment is enhanced; experiments show that in various image and video benchmark tests, compared with an existing method, the framework has the advantages that the calculation cost is greatly reduced, the model performance is maintained and even improved, and the framework has remarkable technical advantages and application potential.
Owner:ZHEJIANG YOULU ROBOT TECH CO LTD

Smart sensing for pallet loading and unloading

A pallet loading system may comprise a processor, and memory with instructions stored thereon that, when executed by the processor, cause the processor to receive package loading data, the package loading data including characteristics of a package to be placed on the pallet and characteristics of a loading operator. A placement location for the package on the pallet may be determined using an artificial intelligence (AI) or machine learning (ML) algorithm based on the characteristics of the package and the characteristics of the loading operator and a visual marking (such as a visual projection) may be displayed at the placement location. The system may output an instruction to the loading operator to place the package at the displayed placement location.
Owner:INTEL CORP

Apparatus and method for locating a position of an electrode on an organ model

An apparatus and method for locating a position of an electrode on an organ model. The apparatus includes a memory communicatively connected to at least a processor, wherein the memory contains instructions configuring the at least a processor to receive an organ model configured to digitally represent an organ, receive a set of sensor data from at least a sensor including an ultrasound sensor, determine an electrode position within the organ model as a function of the set of sensor data using a position machine-learning module, wherein determining the electrode position includes determining a model position within the organ model as a function of the set of sensor data and determining the electrode position within the model position of the organ model as a function of the set of sensor data and add a visual marker onto the electrode position in the model position of the organ model.
Owner:NFERENCE INC

Secured transactions on wearable devices

Systems herein describe a secured transaction system for accessing a camera stream from a front-facing camera unit of a wearable computing device, identifying a visual marker in the camera feed associated with a payment application, in response to identifying the visual marker, transmitting a request to an operating system of the wearable computing device to isolate access of the camera stream to the payment application and in response to transmitting the request, initiating a transaction using the payment application.
Owner:SNAP INC

Historical building brick wall surface damage intelligent identification and diagnosis method

The invention relates to the technical field of historical building protection, and discloses an intelligent identification and diagnosis method for historical building brick wall surface damage, which breaks through the limitations of low efficiency, strong subjectivity and easy detail omission of traditional manual exploration by fusing unmanned aerial vehicle high-precision image acquisition, deep learning semantic segmentation and multi-modal feature analysis technologies. And automatic rapid positioning and accurate classification of large-range wall surface damage are realized. Environmental distortion is eliminated by using a self-adaptive image registration algorithm, fine texture features are extracted in combination with a damage probability prediction model, and complex damage forms such as salting-out crystallization, plant root erosion and weathering spalling are effectively distinguished; through three-dimensional space clustering and visual marking technologies, the damage distribution density and the evolution trend are visually presented, and a reliable basis is provided for scientifically formulating a grading repair strategy. Besides, through establishment of a damage database and iterative optimization of an intelligent diagnosis model, the building health condition can be tracked for a long time, and technical support is provided for preventive protection and sustainable management of cultural heritage.
Owner:SHANGHAI MINGYUE ARCHITECTURAL DESIGN OFFICE CO LTD

Controlling agents using tokenized goal images

PCT designated stage expiredWO2024261187A1Biological modelsPattern recognitionVisual marking
Methods and systems for controlling agents, e.g., robots, using tokenized goal images. One of the methods includes receiving a goal image; tokenizing the goal image to generate a plurality of visual tokens; at each of a plurality of time steps: obtaining one or more observation images characterizing a state of the environment at the time step; tokenizing each of the one or more observation images; generating a sequence of input tokens that comprises the plurality of visual tokens that represent the goal image and the plurality of visual tokens that represent the one or more observation images; processing the sequence of input tokens to generate an output sequence of output tokens from the discrete vocabulary of tokens that represents an action to be performed by the agent in response to the observation images; and causing the agent to perform the selected action.
Owner:DEEPMIND TECH LTD

Automatic driving space planning enhancement method based on visual marker

The invention discloses an automatic driving space planning enhancement method based on a visual marker. The method comprises the following steps: acquiring an original image and text input; processing the original image to obtain image features; processing the text input to obtain text features; generating a text output with a visual mark by using the image features and the text features; the text output with the visual marks is converted, and text output with coordinates is obtained; the accuracy and semantic consistency of spatial understanding in an automatic driving scene are remarkably improved, high synchronization of visual perception and semantic expression is achieved, and the problem of visual and language modal semantic segmentation in an existing method is effectively solved. The analysis precision of the object position, the motion state and the interaction relation in the automatic driving question and answer task is greatly improved, and the decision reliability and the planning naturalness in a complex driving scene can be remarkably enhanced.
Owner:SOUTH CHINA UNIV OF TECH

Positioning control strategy for automatic charging process of Mecanum wheel mobile chassis based on IMU and camera

The invention discloses a positioning control strategy for an automatic charging process of a Mecanum wheel mobile chassis based on an IMU and a camera, and belongs to the field of automatic control. The method comprises the following steps: firstly, marking a visual mark on a charging pile, adjusting a chassis to a specified angle by reading angle information of an inertial measurement unit (IMU), and adjusting the transverse displacement of the chassis to a specified position by reading a pixel coordinate of a central point of the visual mark (such as a color block) marked in a camera; and establishing a local coordinate system and a global coordinate system of the chassis according to the adjusted pose, further solving a local pose and a global pose by combining an IMU (Inertial Measurement Unit), then solving a chassis target speed according to the local pose and the global pose of the chassis and a chassis speed set value, and finally solving the rotating speed of each wheel according to a mobile chassis inverse kinematics model. Therefore, the vehicle is driven to move towards the target charging pile. The invention provides an automatic charging positioning control strategy based on the IMU and the camera, and the vehicle can be accurately moved to the position of the charging pile, so that the automatic charging function of the vehicle is realized.
Owner:ADVANTAGE ZHIYUN (HANGZHOU) TECH CO LTD

Handheld Input Devices

A system may include an electronic device such as a head-mounted device and a handheld input device for controlling the electronic device. A lanyard may be removably attached to the handheld input device or a non-electronic object. The lanyard may include visual markers, such as infrared light-emitting diodes and / or fiducials, that can be detected by an external camera and used to track a location of the lanyard. For example, the lanyard may be fabric, and the visual markers may be incorporated into the fabric or attached to the fabric. The lanyard may also include motion sensors, visual-inertial odometry cameras, or other sensors to determine the location of the lanyard. The lanyard may be electrically coupled to the handheld input device, such as to transfer power and / or data. Alternatively, the lanyard may be coupled to a non-electronic object and may include a battery and / or a haptic output component.
Owner:APPLE INC

Adaptive visual mark pruning method and device based on multi-modal large model

The invention discloses an adaptive visual mark pruning method and device based on a multi-modal large model. The method comprises the following steps: calculating two initial distributions of visual marks; initializing an error threshold according to a binary search algorithm; using a greedy algorithm to delete the visual marks which have the minimum influence on the two initial distributions until the error of the two initial distributions reaches an error threshold value, obtaining two groups of candidate deletion marks, taking an intersection, and deleting the visual marks based on the intersection; updating an error threshold boundary according to the model calculation amount after the visual mark is deleted and the target calculation amount; repeating the deleting and updating steps until a stopping condition is met, and calculating the average deleting number of the visual marks in each layer of a batch of data as a deleting strategy; and sorting the visual marks to be deleted and then deleting the visual marks according to a deletion strategy. According to the method, the use of computing resources is optimized by automatically identifying and removing redundant visual marks, the operation cost is reduced on the premise of not sacrificing the performance of the model, and the reasoning speed of the model is increased.
Owner:XIAMEN UNIV

Image recognition method and system for stone slab crack detection

The invention discloses an image recognition method and system for stone slab crack detection, and the method comprises the steps: carrying out the preprocessing of a color image of a stone slab, and obtaining a gray image; calculating the gradient amplitude of the grayscale image, determining an optimal threshold value, and obtaining a preliminary edge image; calculating the first order gradient and direction of the preliminary edge image, constructing a Hessian matrix and generating a crack response image; performing edge detection on the crack response image to generate an initial edge image; performing connected domain analysis on the initial edge image to generate a final edge image; and extracting coordinates of crack points from the final edge image, mapping the crack points to the stone slab color image, and performing visual marking to generate a marked image. According to the method, high-precision extraction and fine distinguishing of the cracks under the background of complex stone textures are achieved, the limitation that the cracks and the patterns are difficult to separate in a traditional method is effectively overcome, meanwhile, crack marking is directly carried out on the stone slab original drawing, and the accuracy and practicability of stone slab quality detection are remarkably improved.
Owner:HUBEI PROVINCE HUAJIAN STONE CO LTD

FTU-based power distribution network fault positioning method, apparatus and device, and storage medium

The invention relates to an FTU-based power distribution network fault positioning method, which comprises the following steps of performing electrical signal acquisition on a power distribution network through an FTU to obtain an electrical acquisition signal, and performing digital processing on the electrical acquisition signal to obtain an electrical parameter; mapping the electrical parameters to a preset electrical parameter-time coordinate system, and performing curve drawing on the electrical parameters on the electrical parameter-time coordinate system to obtain a visual curve; monitoring whether an abnormal phenomenon exists in the visual curve on the electrical parameter-time coordinate system, and if the abnormal phenomenon exists in the visual curve, marking an abnormal region corresponding to the abnormal phenomenon to obtain a visual marking curve; obtaining the time of an abnormal point based on an abnormal phenomenon on the visual marking curve, and obtaining the fault location of the power distribution network based on the time of the abnormal point and the propagation speed of the electrical acquisition signal; the technical problem that it is difficult to accurately and rapidly determine the fault position under a complex network structure is solved.
Owner:SHENZHEN FRIENDCOM TECH DEV

Network training and object attitude estimation method and device, equipment and storage medium

The embodiment of the invention discloses a network training and object attitude estimation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first sample image containing a preset object and a position mark of a key point of each object in the first sample image; based on the position label of each object key point, shielding at least one image sub-block in the first sample image to obtain a shielded image; utilizing a to-be-trained visual converter network to perform feature extraction on the occlusion image to obtain a first visual mark sequence; using a first decoder network to restore the occluded image sub-blocks in the occluded image based on the first visual mark sequence to obtain a restored image; and updating network parameters of the visual converter network at least once based on a first loss value between the first sample image and the restored image to obtain the trained visual converter network.
Owner:SHANGHAI SENSETIME INTELLIGENT TECH CO LTD

Domain-Specific Shorthand for Generation of Data Visualizations based on Context Free Grammar

System, method and interface for generating data visualizations are provided. The system receives a user input to specify a natural language command directed to a data source. The system also generates a prompt for generating a data visualization based on relevant data fields and data values, rules that characterize the data visualization, and a context free grammar. The system also prompts a trained large language model using the prompt to generate a structured document following a domain-specific schema based on a shorthand notation. The system also uses a parser that uses the context free grammar to map the structured document to a visual specification. The visual specification specifies the data source, visual variables, and data fields from the data source. The system also generates and displaying a data visualization based on the visual specification, including displaying visual marks representing data, retrieved from the data source, for the data fields.
Owner:SALESFORCE INC

Railway tunnel construction crack detection method based on machine vision

The invention relates to a railway tunnel construction crack detection method based on machine vision, which is applied to the field of tunnel detection, and comprises the following steps: marking a crack by using the flight coordinate of an inspection unmanned aerial vehicle body on the basis of the existing unmanned aerial vehicle detection, and then controlling the inspection unmanned aerial vehicle body to approach and confirm; then, the marking bomb is ejected to the crack in a manual control mode, so that visual marking is carried out, researchers can know crack information through data and can rapidly and accurately find the crack through marking to carry out field research and analysis, and therefore the detection efficiency and accuracy are effectively improved; in addition, the method can effectively avoid defects of inspection robots for top lining detection in the prior art, and effectively reduce interference of factors such as distance, dust and vibration.
Owner:发腾实业(云南)有限责任公司

Customer service information generation method and system based on multi-modal intention recognition

The invention discloses a customer service information generation method and system based on multi-modal intention recognition, and the method comprises the steps: obtaining customer service session data, and carrying out the heterogeneous data classification operation of the session data; performing feature extraction operation on each piece of parting modal information; establishing a semantic bridging matrix among the modal features, performing hierarchical attention routing, and outputting an intention label; a knowledge graph is injected, the knowledge graph comprises a static knowledge base, a real-time service flow and a user historical portrait, and three-source dynamic knowledge is output; constructing a decision tree based on the intention label, the three-source dynamic knowledge and the space-time identifier; and executing a corresponding text generation operation, a visual mark generation operation or a voice synthesis operation based on a response action corresponding to the leaf node of the decision tree, and generating customer service information. According to the method and the system, the buyer intention recognition accuracy of the multi-modal session data can be greatly improved, and the response delay time of generating the customer service information is shortened.
Owner:深圳乐搏科技有限公司

Aircraft positioning method and device, aircraft, medium and program product

The embodiment of the invention provides an aircraft positioning method and device, an aircraft, a medium and a program product, and relates to the technical field of flight positioning. The method comprises the following steps: acquiring a to-be-identified image, and identifying a visual marking object; matching and associating the two-dimensional coordinate information corresponding to each visual marking object with the actual three-dimensional coordinate information; wherein each piece of actual three-dimensional coordinate information is obtained based on a preset coordinate transformation rule and is represented by a unified transformation matrix parameter; and constructing and solving an objective function about the two-dimensional coordinate information, the actual three-dimensional coordinate information and the to-be-solved aircraft pose, and obtaining the aircraft pose information according to a solving result. According to the embodiment of the invention, the position information of each mark is represented through the transformation matrix based on the unified coordinate system, so that different marks can be arbitrarily arranged in different planes, and the application flexibility of a visual mark positioning technology in a complex scene is effectively improved.
Owner:TIANJIN YUNSHENG INTELLIGENT TECH CO LTD

Gas-liquid two-phase slug flow related flow measuring device and method based on vibration vision

The invention relates to flow measurement, in particular to a gas-liquid two-phase slug flow related flow measurement device and method based on vibration vision, and aims to solve the problem that an existing two-phase flow meter cannot measure the flow working condition of slug flow with fluctuation characteristics. According to the device, a vibration probe is fixed on a pipeline through a base, a vibration needle is fixed in the base through a sealing piece, one end of the vibration needle extends into the pipeline to detect slug flow fluctuation, a magnet is arranged at the other end of the vibration needle and right faces a magnetic visual mark, magnetic force exists between the magnet and the visual mark, and the magnetic force enables the visual mark and the vibration needle to vibrate synchronously. Based on a dynamic image or video of vibration of the visual mark, a vibration signal of the visual mark can be obtained; on-line measurement of gas-liquid two-phase dynamic flow can be realized by utilizing vibration signals of visual marks corresponding to the double vibration probes positioned on the upstream and downstream of the slug flow, so that flow measurement of the slug flow is not influenced by liquid phase viscosity on probe signals and is not limited by whether the liquid phase is conductive or not, light guide and the like, and the device and the method are suitable for water-gas two-phase flow and can also be used for measuring the flow of the slug flow. And the device is also suitable for oil-gas two-phase flow.
Owner:XI AN JIAOTONG UNIV

Second-order few-sample medical image classification method and system based on multi-mode prompt learning, storage medium and equipment

The invention belongs to the technical field of computer vision image processing, and provides a second-order few-sample medical image classification method and system based on multi-modal prompt learning, a storage medium and equipment, and the method comprises the steps: obtaining few-sample medical image data, introducing auxiliary text input to further describe an image or concept category, promoting cross-modal learning, and carrying out multi-modal prompt learning. Two classification heads are combined: one is a shared classification head for processing image category marks in a visual encoder and prompt representations encoded by a text encoder; another is to classify a visual marker feature distribution from a visual encoder by aggregating visual markers using global covariance pooling with efficient matrix power normalization. According to the method, the classification precision of the few-sample medical images is remarkably improved.
Owner:DALIAN UNIV OF TECH

Visualization of external audio commands

This disclosure relates to methods, systems, and techniques for visualizing external audio captured by a microphone of the vehicle. Using the techniques described herein, external audio signals may be interpreted and converted into clear visual indicium that can be understood by individuals associated with the vehicle, such as passengers inside the vehicle or individuals awaiting pickup.
Owner:ZOOX INC

Video space-time understanding method and device based on multi-modal large model, and medium

The invention belongs to the field of computer vision, and particularly relates to a video space-time understanding method and device based on a multi-modal large model and a medium. A multi-modal large language model is connected with a mask segmentation model, and video features are encoded by using a multi-modal encoder; the time tasks and the space tasks are represented by adopting different numbers of sampling frames and visual mark forms. The visual mark is aligned to the text space and then input into the large language model together with the text mark, and a corresponding text answer is obtained through decoding. For a time task, a timestamp is directly extracted from a text answer, and spatial information passes through lt; sEGgt, SEGgt; and the mark code is embedded as a prompt input to the mask decoder, so that mask generation of the sampling frames and mask propagation of the whole video are realized. Compared with the prior art, the method has the advantages that joint training of fine-grained video time-space understanding is realized, and more accurate event time-space positioning can be realized besides overall understanding of the video.
Owner:FUDAN UNIVERSITY