269 results about "Visual cueing" patented technology

The invention provides a zero sample anomaly detection method and device and electronic equipment, and relates to the technical field of image anomaly detection.The method comprises the steps that a to-be-detected image is acquired, and visual features are extracted through a CLIP model; performing visual enhancement on the visual features to obtain enhanced visual features; injecting the enhanced visual features into a learnable text prompt template to generate an adaptive text prompt; injecting the adaptive text prompt into a text encoder for encoding to obtain a text embedding representation; mapping the adaptive text prompt to a visual space to obtain a visual prompt, and inputting the visual prompt into the local visual features to obtain scale visual features; and based on the text embedding representation and the scale visual features, separately calculating an anomaly score and an anomaly positioning map of a preset scale, and fusing to obtain a detection result. According to the zero sample anomaly detection method and device and the electronic equipment provided by the invention, the generalization ability of the whole detection process is effectively improved, and the actual application requirements are further met.

A computer system, method, and program product facilitate human-computer interaction. A processor set receives a non-text visual cue and natural language instruction regarding a scene. The processor set converts the non-text visual cue into a textual location information indicating a portion of an image representing the scene. A language machine learning model is triggered by using the textual location information, the natural language instruction, and the image representing the scene as input. The language machine learning model outputs a response to the input.

The invention provides an experiment task execution method and device.The method comprises the steps that task information of a target experiment task is divided based on a visual language model, a subtask sequence is obtained, and the subtask sequence is formed by arranging multiple subtasks from front to back according to the execution sequence; processing the text information corresponding to the current sub-task and the experiment image before the current sub-task is executed through the visual language model from the first sub-task in the sub-task sequence to obtain a visual prompt image, and executing the visual prompt image through a visual language action model. And processing based on the text information, the experiment image and the visual prompt image, guiding a robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, and determining that the target experiment task is completed. According to the method, the success rate, the operation safety and the regulation compliance of the experiment task are effectively improved, and the method has high universality and safety.

The invention provides a remote sensing image target statistical method and system fusing a large language model and visual prompt driving. The method comprises the following steps: acquiring a remote sensing instance segmentation image to be processed and a visual prompt thereof; inputting a to-be-processed remote sensing instance segmentation image and a visual prompt thereof into the trained remote sensing image target statistical model, and outputting a remote sensing image target statistical result; the training comprises the following steps: introducing a large language model and visual cue into an encoder architecture of a GrondingDINO model to obtain a remote sensing image target statistical model; inputting a remote sensing instance segmented image and the visual cue thereof into an encoder, and outputting an image feature, a visual cue feature and a text feature; the feature intensifier carries out fusion processing on the output of the encoder; a language-guided query selection module calculates cross-modal query according to the fusion processing result, and a cross-modal decoder obtains a target statistical result of the image based on the fusion processing result and the cross-modal query; and training by using the training data and outputting the trained model.

The invention discloses an unsupervised continuous anomaly detection method and system for multi-mode prompt memory, and relates to the technical field of computer vision and industrial image detection. The method comprises the following steps: acquiring industrial images to form a data set; an unsupervised anomaly detection model is constructed, the unsupervised anomaly detection model comprises a visual branch and a text branch which are respectively used for extracting visual features and text features, learnable visual prompts and learnable text prompts are introduced into the unsupervised anomaly detection model, and then the visual features and the text features are fused by using a self-adaptive fusion mechanism to obtain an anomaly detection result; training learnable visual prompts and learnable text prompts in the unsupervised anomaly detection model by using the data set; and performing anomaly detection on the industrial data by using the trained unsupervised anomaly detection model. According to the method, multi-modal information can be fused in the learning process, the continuous learning ability is achieved, and efficient unsupervised anomaly detection of industrial products is achieved.

The invention relates to the technical field of medical image processing, in particular to an ultrasonic image segmentation method based on edge guidance, and the method comprises the steps: inputting an ultrasonic image into an edge extraction branch and an image encoder; outputting an edge mask image corresponding to the image through the edge extraction branch; performing noise suppression and topological correction on the edge mask image to generate a closed edge constrained by an anatomical structure; generating prompt box information based on the closed edge, and inputting the prompt box information to a prompt encoder; fusing the image prompt features output by the prompt encoder, the image features extracted by the image encoder and the edge mask information; the fusion features are input into a decoding module, a final segmentation result is obtained, the synergistic effect of edge information and visual prompt is fully utilized, the perception ability of the model for the anatomical structure in the ultrasonic image is improved, and the accuracy and robustness of segmentation are remarkably improved. Therefore, the problems of fuzzy boundary, inaccurate prompt, weak structure identification capability and the like in related technologies are solved.

Implementations relate to dynamically adapting a given assistant output based on a given persona, from among a plurality of disparate personas, assigned to an automated assistant. In some implementations, the given assistant output can be generated and subsequently adapted based on the given persona assigned to the automated assistant. In other implementations, the given assistant output can be generated specific to the given persona and without having to subsequently adapt the given assistant output to the given persona. Notably, the given assistant output can include a stream of textual content to be synthesized for audible presentation to the user, and a stream of visual cues utilized in controlling a display of a client device and / or in controlling a visualized representation of the automated assistant. Various implementations utilize large language models (LLMs), or output previously generated utilizing LLMs, to reflect the given persona in the given assistant output.

Embodiments of the present disclosure provide a method, device, and medium for training a large scale object foundation model. The method comprises obtaining a training dataset comprising a plurality of subsets for a plurality of object perception tasks, wherein a sample in the training dataset comprises an image with an object, a prompt indicating the object, and labeled object perception information of the image. The method further comprises generating, by the image encoder, an image feature based on the image. The method further comprises generating, by the text encoder or the visual prompt encoder, a prompt embedding based on the prompt. The method further comprises generating, by the object decoder, object perception information of the object based on the image feature and the prompt embedding. In addition, the method further comprises training the object processing model based on the generated object perception information and the labeled object perception information.

The present disclosure teaches a keycap with an interchangeable keycap top. The keycap comprises a keycap base and a keycap top. Wherein, a first magnetic piece connects to the keycap base, and a second magnetic piece connects to the keycap top. Both magnetic pieces can be a magnet or of a magnetic material, and at least one of the magnetic pieces is a magnet. The keycap top can thus be secured onto the keycap base by attaching the first magnetic piece to the second magnetic piece. The keycap top may be textured, have a distinctive shape, or have a distinctive coloring, or have other features, to provide physical and visual cues for a user. The present disclosure also teaches a customizable keyboard and a method of customizing such keyboard using said interchangeable keycap tops.

The invention discloses a multi-modal face living body detection method and device based on text enhancement, and the method comprises the steps: inputting a multi-modal face image into an image block embedding module and a visual prompt generation module, and carrying out the encoding through an image encoder, thereby obtaining an image feature and a visual prompt; inputting the dichotomy text description into a text embedding module, randomly initializing a text prompt vector, and encoding through a text encoder to obtain text features and text prompts; respectively enhancing image features and visual prompts by using a hybrid expert module and a bypass prompt enhancement module; performing information exchange on the visual prompt and the text prompt by using a text enhancement module to obtain an enhanced text prompt, and performing information exchange on the image feature and the text feature by using an image mask module to obtain an enhanced image feature; and finally calculating the similarity between the image features and the text features, and taking the category corresponding to the highest similarity as a detection result. According to the invention, the method can effectively enhance the discrimination capability of human face features, and improves the detection accuracy and generalization.

The invention relates to the technical field of intelligent visual detection, in particular to visual field detection equipment, system and method based on extended reality (XR), and the method comprises the steps that an image feature extraction module collects image data in real time, extracts features and eliminates optical distortion; the fixation deviation correction module analyzes eye movement data in real time, quantifies fixation deviation and triggers visual prompt and correction; the adaptive stimulation generation module dynamically encrypts to generate a test point image and adaptively adjusts the test point image according to visual field analysis and test requirements; the dynamic adjustment image module predicts and optimizes a test point presentation sequence according to image features and fixation points, automatically encrypts a high-probability defect area and quickly jumps to a key screening area; and the visual field defect identification module extracts image features, calculates the defect probability of each region in real time by using a Bayesian network in combination with subject response data, and locates a high-risk region. The system provides an efficient and accurate visual field detection scheme through the synergistic effect of multiple modules.

The invention belongs to the technical field of intelligent sports and computer vision crossing. The adaptive training method based on Tai Chi action recognition is provided, and space action features are extracted according to three-dimensional human skeleton key point coordinates of a human body; extracting multi-scale motion features of the human body in a time domain according to the spatial motion features; global attention aggregation is carried out according to the multi-scale motion features of the human body in the time domain, and a human body motion recognition result is obtained; according to a human body action recognition result, matching a corresponding Tai Chi style drawing theory; superposing multi-dimensional visual prompts into a Tai Chi motion video according to the Tai Chi motion style theory so as to intuitively prompt a user about action key points; calculating an action problem based on priori knowledge or a preset template, prompting an error position and an error reason by adopting graphs and / or characters, and giving an adjustment suggestion; according to the invention, more personalized training experience can be provided, cognitive load can be reduced, and training efficiency, training effect and cognitive learning fluency can be improved.

The embodiment of the invention provides a remote sensing target segmentation method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-segmented remote sensing image and a sample library; calculating the feature similarity between the to-be-segmented target semantic tag and each sample, and screening out a target sample of which the feature similarity meets a preset requirement from the sample library; generating a target reference mask based on the target sample; and converting the target reference mask into a visual prompt, and inputting the visual prompt and the to-be-segmented target semantic tag into a segmentation model to obtain a segmentation result. Through combination of sample library assistance and visual prompt guidance, accurate segmentation of a specific target in a remote sensing image is realized, and dependence on dense manual annotation is eliminated. Meanwhile, the generalization ability of the model to the same kind of targets in different scenes is enhanced through double constraints of semantic tags and visual features.

The invention relates to a video action recognition model training method, a video action recognition method and a video action recognition device. The method comprises the following steps: acquiring a sample video frame image and an action category description text corresponding to the sample video frame image, inputting the sample video frame image and the action category description text into a to-be-trained recognition model, and recognizing the action category of the to-be-trained recognition model by an image encoder in the recognition model according to a preset visual prompt vector and the sample video frame image, generating a video embedding corresponding to a sample video frame image, generating a text embedding corresponding to an action category description text by a text encoder in the recognition model based on a preset text prompt vector and the action category description text, and constructing bidirectional comparison loss by taking the video embedding and the text embedding as positive sample pairs, and updating the visual prompt vector and the text prompt vector based on the bidirectional contrast loss to obtain a trained recognition model. By adopting the method, the video action recognition accuracy can be improved.

The invention discloses a high-performance unmanned aerial vehicle detection method based on a visual cue word, and relates to the technical field of unmanned aerial vehicle detection, and the method comprises the steps: collecting diversified images containing a target to form a training data set, collecting a few-sample cross-domain image as cross-domain information input, inputting the training image into a pre-trained and parameter-locked backbone network, and extracting basic features; a few-sample cross-domain image is input into a cross-domain network, after features are extracted, a learnable visual prompt vector is generated, noise is added, then the prompt vector and basic features are input into a neck module through a cross attention mechanism to be integrated to obtain fusion features, and finally a detection head is input to output a classification and position regression result of a target. By reducing the training cost and improving the cross-domain generalization performance of an unmanned aerial vehicle algorithm, an efficient solution is provided for low-resource cross-domain detection tasks such as industrial quality inspection and geographical remote sensing, and meanwhile, a solution is provided for training a high-performance unmanned aerial vehicle through extremely few training parameters.

Embodiments of the present disclosure provide a visual language model training method, an image tag prediction method, and an electronic device. According to the embodiments of the present disclosure, encoding parameters respectively corresponding to an image encoder and a text encoder in a first visual language model are fixed, on the basis of initial values of visual prompts and text samples respectively corresponding to a plurality of image classification tags, a second visual language model is generated by training the first visual language model, and the visual prompts respectively corresponding to the plurality of image classification tags are mined and learned, thereby enhancing the visual representation capabilities of visual language models. Next, on the basis of the learned visual prompts, text prompts, and the text samples, a target visual language model is further trained by training the second visual language model, and adapter parameters of model adapters respectively configured for the image encoder and the text encoder are adjusted, to collaboratively optimize image feature extraction and text feature extraction. This process allows for the transfer of visual knowledge from visual prompts to text prompts.

A computer implemented method for learning a visual prompt on a Multimodal Large Language Model (MLLM) for a downstream task, comprising: applying a visual prompt to a plain image, wherein the visual prompt is a set of parameters in a pixel space; obtaining a first feature embedding of the prompted image and a second feature embedding of the plain image; outputting a textual prediction corresponding to the prompted image for the downstream task by the MLLM based on a projection of the first feature embedding to a text embedding space; and optimizing the visual prompt, with parameters of the MLLM fixed, by minimizing a loss function constructed based at least on a relative entropy loss between the first feature embedding and the second feature embedding. Several other aspects are disclosed.

Battery life cycle management is facilited with distinct visual cues of colors and color-coded combinations. The color-codes enable battery types and characteristics over the course of the life of the battery to be determined. The color-codes are based on a color system that indicates a performance rating of the batteries. A system for managing the color-coded batteries selects color-coded batteries for use based upon color, maintaining color based functionality, and color-coded battery indication for destruction of the battery at the end of its life cycle.

The invention discloses a visual prompt multi-modal large model for multi-source remote sensing image interpretation, which belongs to the technical field of crossing of remote sensing and computer vision and comprises a multi-modal content coding and integration module, a cross-domain first-stage fusion training module, a pixel level visual positioning module and a large language model. The model supports the interpretation of a remote sensing image after the remote sensing image is arbitrarily amplified and reduced, and has flexible multi-granularity vision and language interaction capability. According to the model, a large language model is used as an interface, and multi-modal content integration including multi-sensor images, visual prompts and text instructions is achieved. In addition, two types of space tasks of anaphora understanding and visual positioning are unified into a visual prompt learning framework, and comprehensive and flexible multi-granularity understanding of remote sensing data is promoted.

A method and system for visualizing music using a perceptually conformal mapping system are provided. A music source file is input into a processor configured to carry out a series of steps on audio cues identified within the music and ultimately generate a simultaneous visual representation on a display device. The series of steps include application of one or more perceptually conformal mapping systems that essentially induce a synesthetic experience in which a person can experience music both acoustically and visually at the same time. The device extracts cues from the music that are designed to specifically capture fundamentals of human appreciation, maps them into visual cues, then presents those visual cues synchronized with the source music.

The invention discloses a long video generation method and device, equipment, a readable storage medium and a program product, relates to the technical field of communication, and aims to improve the quality of a generated long video. The method comprises: obtaining a video description text to be processed and video processing parameters, the video processing parameters comprising a video segment number N and a video frame number M included in each video segment, N being an integer greater than or equal to 2, and M being an integer greater than or equal to 1; dividing the video description text into text information corresponding to N time periods; taking the text information corresponding to each time period and the historical video visual prompt word corresponding to each time period as input of a video visual encoder, and running the video visual encoder to obtain a video clip corresponding to each time period; and splicing the video clips to obtain a long video. According to the embodiment of the invention, the quality of the generated long video can be improved.

The disclosure herein describes interpreting attention-based decisions of summarization outputs generated by a deep learning model. A decision interpretation model obtains attention values defining connections between input tokens associated with a source text and output tokens for a selected portion of a summary associated with the source text. The input tokens having the highest attention values indicating the strongest connections between the input tokens of the source text and an output token of the summary are selected as primary tokens. A semantic similarity between the primary tokens for each attention head and an output token is calculated. The model selects the primary tokens having the closest semantic similarity with the summary portion. A visual cue is generated on or within a portion of the source text corresponding to the primary tokens. The visual cue identifies dominant words in the source text used to explain the summary portion.

The invention discloses a target navigation method and device based on active three-dimensional Gaussian spatter and visual language model reasoning, and the method comprises the steps: obtaining an RGB-D image and a pose in an unknown environment, and constructing an incremental three-dimensional Gaussian spatter map as persistent memory through active perception; generating an exploration map based on the constructed 3DGS map, extracting a leading edge point and carrying out space structure adaptive clustering; generating a guidance track, carrying out free viewpoint optimization based on the track, and rendering a leading edge point first-person view angle image containing rich information; constructing a structured visual prompt, combining with a thinking chain prompt, inputting a visual language model to carry out reasoning planning, and selecting an optimal navigation target; in the navigation process, a real-time target detector is used for screening potential targets, and a new view angle is rendered in a 3DGS space through an action decision VLM for target re-verification. The navigation success rate and efficiency are improved.

A method for embodied agent exploration is described. The method includes building a semantic map of a surrounding scene based on depth information and via visual prompting of a vision language model (VLM). The method also includes utilizing conformal prediction to calibrate a question answering confidence of the VLM. The method further includes performing, by an embodied agent, scene exploration utilizing knowledge of relevant regions of the scene. The method also includes determining, by the embodied agent, when to terminate the scene exploration utilizing a calibrated question answering confidence of the VLM.

The invention relates to a method for the situational output of visual cues to support the driving task of a person (6) driving a vehicle (1) on a display device (7). The method according to the invention is characterized in that a position of the vehicle (1) and a representation of the environment related to the current position of the vehicle (1) are determined. An interior camera (2) is used to capture an image of the person (6) driving the vehicle (1), from which image a corneal region with corneal reflex images is extracted. The extracted image region is then transformed into a driver image model and the representation of the environment and projected onto a common projection surface in a common coordinate system. The driver image model and the representation of the environment are reduced to the extent of the smaller of the two models, after which the representation of the environment and the driver image model are registered.wherein differences between the representation of the environment and the driver image model are determined, and then, depending on the number of differences, an adjustment of optical indications superimposed on the display device (7) of the environment is carried out.

A method and apparatus enables modifying the electronic controls of EVs to mimic the sensory experience of driving a performance ICE car. The method and apparatus creates a sensory “virtual cockpit” with both electronic and mechanical enhancements for a sensory experience. By downloading and implementing the method and apparatus, one may mimic, for example, the vehicle dynamics, performance horsepower, torque curves, suspension settings, oversteer and understeer behavior, steering-wheel inputs, cabin sound, subtle cabin vibrations, and audio / visual cues via a graphical user interface. These simulations replicate, in an electric vehicle, the various aspects of an ICE vehicle to mimic the whole experience of driving various ICE performance vehicles.

An autonomous herding system using a fleet of vehicles to manage and retrieve stray animals, combining technologies from U.S. Patents US 12102060 and US 12153451. The system employs autonomous vehicles equipped with optical-LiDAR sensors, GPS, and machine-learning image recognition to detect and guide strays back to the herd. Vehicles operate in two modes: roundup, for gathering scattered animals, and herding, for maintaining cohesion and directing movement. Positioned in traditional formations (point, swing, flank, drag), they use strategic movements, visual cues, and audio signals to influence herd behavior based on flight zone and point-of-balance principles. A trail boss module coordinates vehicle actions, ensuring efficient herd control. The system includes a battery replacement station for continuous operation, automatically swapping depleted batteries. This integration streamlines livestock management, reducing labor costs while improving animal and handler safety.

The invention discloses a training method and system for a small amount of multi-modal pathological annotation data mixed prompt learning framework. The training method comprises the steps of obtaining an original CLIP model, an initial text prompt and a training image corresponding to the initial text prompt, wherein the original CLIP model comprises an image encoder and a text encoder; inserting a learnable visual prompt mark in the image encoder, and inserting a learnable text prompt mark in the text encoder; inputting the training image into an image encoder and combining the training image with a learnable visual prompt mark for feature coding to obtain visual prompt embedding; inputting the initial text prompt into a text encoder and combining the initial text prompt with a learnable text prompt mark for feature coding to obtain text prompt embedding; on the basis of the similarity of visual prompt embedding and text prompt embedding, parameters of learnable visual prompt marks, learnable text prompt marks and coupling functions are adjusted, a mixed prompt learning framework is obtained, and the multi-modal combination capacity of the model is improved.

The invention discloses a feature fusion optimization-based multi-modal map enhancement retrieval method and a dialogue system. The method comprises the following steps of: respectively carrying out pre-training and fine tuning on a visual model and a language model by utilizing a domain image and text data; constructing a knowledge graph based on the text data in the knowledge base and constructing a vector database containing associated image data; performing semantic analysis and optimization on the original query of the user by using the language model and forming a structured retrieval intention; searching related sub-graphs, text semantic vector information and associated image data based on the search intention; encoding the sub-images into knowledge contexts, inputting the knowledge contexts into a dynamic prompt generator to generate visual prompts, and extracting enhanced visual features from the associated image data through a visual model; and inputting the subgraph, the text semantic vector information and the enhanced visual features into a language model for collaborative reasoning, and generating and outputting a final answer. According to the method, deep fusion and accurate retrieval of multi-modal knowledge can be realized, and the accuracy and efficiency are remarkably improved.

The invention provides a multi-task non-ideal measurement CT image reconstruction method and system, equipment and a medium, and designs a text-vision collaborative prompt contrast learning method which is used for multi-task non-ideal measurement CT reconstruction. In order to integrate text and visual features in the network, a text-visual collaborative prompt module is designed, and the module combines semantic representations of different text prompts with fine-grained features of visual prompts, so that the controllability, interpretability and degradation adaptability of a CT image reconstruction process are enhanced. According to the invention, the high-frequency enhancement module is constructed as a core component of a main network structure, and the module significantly improves the extraction and optimization of the network on high-frequency information through a self-attention mechanism and an adaptive filtering mechanism, thereby effectively relieving the loss problem of high-frequency details. According to the method, a novel composite loss function is introduced, and the visual quality and fidelity of a reconstructed image can be remarkably improved.