360 results about "Visual cueing" patented technology

The invention discloses a visible light and infrared image fusion target tracking method based on visual cue learning, and relates to the field of extended target tracking. Comprising the steps of inputting video image data, converting an image through a block embedding module, extracting multi-scale information through scale perception attention, extracting each modal feature through a double-branch backbone network, performing specific modal visual prompt learning and inter-modal visual prompt interaction, and performing cross fusion of an adapter and a prediction head. According to the method, a more reliable and efficient target tracking system is realized by introducing a multi-scale information extraction mechanism, enhancing visual prompt learning and interaction capability and developing a more advanced multi-modal fusion strategy, and through the improvements, the performance of target tracking in a complex environment can be remarkably improved, and the requirements of practical application are met.

The invention provides a zero sample anomaly detection method and device and electronic equipment, and relates to the technical field of image anomaly detection.The method comprises the steps that a to-be-detected image is acquired, and visual features are extracted through a CLIP model; performing visual enhancement on the visual features to obtain enhanced visual features; injecting the enhanced visual features into a learnable text prompt template to generate an adaptive text prompt; injecting the adaptive text prompt into a text encoder for encoding to obtain a text embedding representation; mapping the adaptive text prompt to a visual space to obtain a visual prompt, and inputting the visual prompt into the local visual features to obtain scale visual features; and based on the text embedding representation and the scale visual features, separately calculating an anomaly score and an anomaly positioning map of a preset scale, and fusing to obtain a detection result. According to the zero sample anomaly detection method and device and the electronic equipment provided by the invention, the generalization ability of the whole detection process is effectively improved, and the actual application requirements are further met.

The invention relates to a steel bridge disease detection and identification method based on a large language model, and belongs to the technical field of artificial intelligence and civil engineering crossing. According to the method, a cross-modal feature alignment mechanism is constructed through a pre-trained multi-modal large language model by fusing a steel bridge image and a field customized text prompt, and a cascade detection process of'component identification-disease classification-region segmentation 'is realized. Comprising the following steps: designing a structured text prompt word bank to enhance semantic consistency, and dynamically fusing general knowledge and instance features in combination with a mixed prompt mechanism; a multi-level cross-modal alignment strategy is adopted to generate an anomaly graph, and a disease area is accurately positioned; a visual prompt enhancement module is introduced to improve the multi-scale feature discrimination ability, and the robustness in a complex environment is adjusted and optimized through data self-adaption. Under the condition of few samples or even zero samples, high-sensitivity detection and pixel-level segmentation of steel bridge cracks, corrosion and other diseases are achieved, and the problems that a traditional method is low in efficiency, poor in generalization, high in labor cost and the like are effectively solved.

The embodiment of the invention provides a sample set construction method, a question and answer model training method, a question and answer processing method, a request processing method and a task platform, and the sample set construction method comprises the steps: inputting a sample image into an image annotation model, and obtaining a global annotation result of the sample image; identifying a target object in the sample image, marking the target object in the sample image based on an obtained identification result to obtain a global view, and intercepting the target object to obtain a local view; the global view, the local view and visual prompt information are input into the image annotation model, a local annotation result of the sample image is obtained, and the visual prompt information is used for prompting the image annotation model to execute an annotation task on the global view based on the local view; and constructing a sample set based on the sample image, the global labeling result and the local labeling result. According to the method, the accuracy and the comprehensiveness of the annotation result of the image can be improved.

The invention provides a method and system for efficiently marking and segmenting a focus area in a medical image, and the method comprises the steps: S1, collecting a medical image data set, and dividing the medical image data set into a training set and a test set; s2, selecting a reference image from the training set, carrying out initial labeling, generating a visual prompt, and obtaining the reference image with the visual prompt; s3, constructing a deep convolutional neural network architecture, and obtaining a trained model through model training; s4, performing prediction segmentation on the unlabeled image in the test set by using the trained model to generate a prediction segmentation result; and S5, checking and correcting the prediction segmentation result, and improving the model performance through interactive optimization. According to the method, a doctor only needs to carry out initial labeling on a small number of focus areas, the problem that an existing fully-supervised segmentation method seriously depends on a large amount of high-quality labeling data is solved, the requirement for the high-quality labeling data is remarkably reduced, and the workload of medical staff is relieved.

The invention discloses a product defect unsupervised semantic segmentation method and device based on a pre-training model, and is used for solving the technical problem that the defect classification precision is not ideal due to an existing industrial defect unsupervised semantic segmentation method. The method comprises the following steps: acquiring a to-be-detected industrial product image, an industrial product reference image corresponding to the to-be-detected industrial product image and a plurality of to-be-matched defect categories; a preset segmentation network of a self-learning point generation prompter based on defect potential space features is adopted to generate a target defect segmentation mask according to a to-be-detected industrial product image and an industrial product reference image, and a target cutting image is generated based on the target defect segmentation mask, so that the target cutting image with visual prompt can be generated; and processing the target cutting image and a plurality of defect categories to be matched through a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module, and combining segmentation and classification, thereby improving the defect classification precision.

A computer system, method, and program product facilitate human-computer interaction. A processor set receives a non-text visual cue and natural language instruction regarding a scene. The processor set converts the non-text visual cue into a textual location information indicating a portion of an image representing the scene. A language machine learning model is triggered by using the textual location information, the natural language instruction, and the image representing the scene as input. The language machine learning model outputs a response to the input.

The invention provides an experiment task execution method and device.The method comprises the steps that task information of a target experiment task is divided based on a visual language model, a subtask sequence is obtained, and the subtask sequence is formed by arranging multiple subtasks from front to back according to the execution sequence; processing the text information corresponding to the current sub-task and the experiment image before the current sub-task is executed through the visual language model from the first sub-task in the sub-task sequence to obtain a visual prompt image, and executing the visual prompt image through a visual language action model. And processing based on the text information, the experiment image and the visual prompt image, guiding a robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, and determining that the target experiment task is completed. According to the method, the success rate, the operation safety and the regulation compliance of the experiment task are effectively improved, and the method has high universality and safety.

The invention discloses a remote sensing image change detection method based on fine tuning CLIP, and the method comprises the steps: obtaining an original dual-time remote sensing image, and carrying out the preprocessing of the remote sensing image; constructing an initial remote sensing image change detection model, and training the initial remote sensing image change detection model by using the training set to obtain a target remote sensing image change detection model; and performing change detection on the remote sensing image by using the target remote sensing image change detection model. According to the method, a large model CLIP pre-trained in a large-scale data set is introduced into a change detection task, and the method aims at guiding the model to pay attention to universal features of changing ground features, so that the generalization ability of the model is improved; according to the method, the CLIP is finely adjusted by adopting a visual prompt method, so that the CLIP can be well migrated in a remote sensing downstream task; in addition, when a loss function is constructed, Dice coefficient loss is introduced to enhance the sensitivity of the model to a positive example.

The method involves using a digital badge device worn on the chest of a user to visually convey personal attributes to others in line of sight. It begins by configuring the badge device to communicate wirelessly with the user's mobile device. Next, a personal attribute of the user is determined based on input from the user or device-captured data. This attribute could relate to the user's affiliation, emotional state, physical activity, and more. Finally, a pre-defined color and / or symbol code representing the determined attribute is displayed on the badge device's screen for others to see. This provides an easily interpretable visual cue to communicate specific personal attributes to different users. By employing this method, the badge wearer can convey information about themselves without the need for verbal communication. The method offers a convenient and efficient way to signal personal attributes in situations where direct interaction may be limited or impractical.

The invention provides a remote sensing image target statistical method and system fusing a large language model and visual prompt driving. The method comprises the following steps: acquiring a remote sensing instance segmentation image to be processed and a visual prompt thereof; inputting a to-be-processed remote sensing instance segmentation image and a visual prompt thereof into the trained remote sensing image target statistical model, and outputting a remote sensing image target statistical result; the training comprises the following steps: introducing a large language model and visual cue into an encoder architecture of a GrondingDINO model to obtain a remote sensing image target statistical model; inputting a remote sensing instance segmented image and the visual cue thereof into an encoder, and outputting an image feature, a visual cue feature and a text feature; the feature intensifier carries out fusion processing on the output of the encoder; a language-guided query selection module calculates cross-modal query according to the fusion processing result, and a cross-modal decoder obtains a target statistical result of the image based on the fusion processing result and the cross-modal query; and training by using the training data and outputting the trained model.

The invention discloses a video understanding method and device based on a multi-modal large model and a medium, and the method comprises the steps: carrying out the frame-by-frame fusion of visual prompt information corresponding to a predetermined first frame and a video frame sequence through a dynamic Alpha mixing technology; extracting multi-frame features from the fused video frame sequence, and integrating the multi-frame features to generate corresponding space-time visual representation; for each current frame in the video frame sequence, fusing the visual prompt information of the previous frame corresponding to the current frame with the visual features of the current frame, and generating visual prompt information of all frames after the first frame in the video frame sequence; and according to the dynamic Alpha mixed coefficient, performing smooth transition on the visual prompt information of every two adjacent frames in the video frame sequence so as to generate a corresponding video understanding result through an autoregression language model in combination with the text instruction and the space-time visual representation.

The invention discloses an unsupervised continuous anomaly detection method and system for multi-mode prompt memory, and relates to the technical field of computer vision and industrial image detection. The method comprises the following steps: acquiring industrial images to form a data set; an unsupervised anomaly detection model is constructed, the unsupervised anomaly detection model comprises a visual branch and a text branch which are respectively used for extracting visual features and text features, learnable visual prompts and learnable text prompts are introduced into the unsupervised anomaly detection model, and then the visual features and the text features are fused by using a self-adaptive fusion mechanism to obtain an anomaly detection result; training learnable visual prompts and learnable text prompts in the unsupervised anomaly detection model by using the data set; and performing anomaly detection on the industrial data by using the trained unsupervised anomaly detection model. According to the method, multi-modal information can be fused in the learning process, the continuous learning ability is achieved, and efficient unsupervised anomaly detection of industrial products is achieved.

The invention provides a carsickness relieving method and system based on display screen dynamic background image enhancement, and the method comprises the steps: collecting the acceleration and posture data of a vehicle through a sensor, and carrying out the real-time fusion processing of the data through a data fusion algorithm; the input display picture is cut and zoomed, a black and white stripe area is set, black and white stripes are controlled to move, the rotation angle of an effective display area is adjusted in real time through an algorithm by means of sensor data, black spots are set in the effective display area to move up and down for visual compensation and other processing, and video signals are transcoded and processed; and the video content is output to an adaptive display interface, so that the display of the video content is realized, carsickness is relieved by introducing visual prompts to passengers in the vehicle, and the comfort and the sense of safety of the passengers are improved.

The invention discloses a passenger car component fault detection method and system based on visual prompt and a visual large model, and the method comprises the steps: carrying out the position marking of a target in each sample image for each type of component sample image set, splicing a panoramic image, and generating a mask image; describing the exception type of each type of part and the details of each exception type through the exception description text; extracting multi-scale image features of the panoramic image and the mask image, and fusing the multi-scale image features to obtain visual prompt features; extracting a multi-scale image feature of the to-be-detected part image, fusing the multi-scale image feature with the visual prompt feature to obtain a multi-scale image feature of the to-be-detected part, inputting the multi-scale image feature into the part positioning model, and positioning a target part image; and fusing the visual features of the target part image with the text features of the corresponding abnormal description text by using a visual large model, and predicting and outputting a fault detection result. According to the invention, precise part positioning and fault detection can be realized.

One or more light sources may be integrated with a medication delivery device, and states of the light sources may be changed. For example, a first lighting state change may be performed, such as based on a detection that an applied force meets or exceeds a threshold break force or an interaction of a movable component with a proximal sensor integrated with the medication delivery device. The first lighting state change may provide visual feedback indicating that an injection has begun. Additionally, a second lighting state change may be performed, such as based on a detection that a resistive force meets or exceeds a threshold back force or an interaction of the movable component with a distal sensor integrated with the medication delivery device. The second lighting state change may indicate that a complete dosage of the medication has been delivered from the medication delivery device.

The invention provides a method and a device for selectively positioning a sound source based on visual prompt, a medium and a product. The method comprises the following steps: acquiring a mixed audio signal and a prompt image; the mixed audio signal comprises audio signals corresponding to different sound events triggered by at least two different sound sources; the prompt image is associated with a target sound source, and the target sound source is a required sound source selected and positioned from different sound sources; the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event; processing the mixed audio signal and the prompt image through a preset cross-instance audio-visual positioning model, including estimating a target mask; and outputting a target arrival direction corresponding to the target sound source based on the target mask. According to the invention, the specific sound source corresponding to the specific sound event is selected and positioned from a plurality of different sound events triggered by a plurality of sound sources, and the application range is wide.

The invention relates to the technical field of medical image processing, in particular to an ultrasonic image segmentation method based on edge guidance, and the method comprises the steps: inputting an ultrasonic image into an edge extraction branch and an image encoder; outputting an edge mask image corresponding to the image through the edge extraction branch; performing noise suppression and topological correction on the edge mask image to generate a closed edge constrained by an anatomical structure; generating prompt box information based on the closed edge, and inputting the prompt box information to a prompt encoder; fusing the image prompt features output by the prompt encoder, the image features extracted by the image encoder and the edge mask information; the fusion features are input into a decoding module, a final segmentation result is obtained, the synergistic effect of edge information and visual prompt is fully utilized, the perception ability of the model for the anatomical structure in the ultrasonic image is improved, and the accuracy and robustness of segmentation are remarkably improved. Therefore, the problems of fuzzy boundary, inaccurate prompt, weak structure identification capability and the like in related technologies are solved.

Implementations relate to dynamically adapting a given assistant output based on a given persona, from among a plurality of disparate personas, assigned to an automated assistant. In some implementations, the given assistant output can be generated and subsequently adapted based on the given persona assigned to the automated assistant. In other implementations, the given assistant output can be generated specific to the given persona and without having to subsequently adapt the given assistant output to the given persona. Notably, the given assistant output can include a stream of textual content to be synthesized for audible presentation to the user, and a stream of visual cues utilized in controlling a display of a client device and / or in controlling a visualized representation of the automated assistant. Various implementations utilize large language models (LLMs), or output previously generated utilizing LLMs, to reflect the given persona in the given assistant output.

Embodiments of the present disclosure provide a method, device, and medium for training a large scale object foundation model. The method comprises obtaining a training dataset comprising a plurality of subsets for a plurality of object perception tasks, wherein a sample in the training dataset comprises an image with an object, a prompt indicating the object, and labeled object perception information of the image. The method further comprises generating, by the image encoder, an image feature based on the image. The method further comprises generating, by the text encoder or the visual prompt encoder, a prompt embedding based on the prompt. The method further comprises generating, by the object decoder, object perception information of the object based on the image feature and the prompt embedding. In addition, the method further comprises training the object processing model based on the generated object perception information and the labeled object perception information.

The present disclosure teaches a keycap with an interchangeable keycap top. The keycap comprises a keycap base and a keycap top. Wherein, a first magnetic piece connects to the keycap base, and a second magnetic piece connects to the keycap top. Both magnetic pieces can be a magnet or of a magnetic material, and at least one of the magnetic pieces is a magnet. The keycap top can thus be secured onto the keycap base by attaching the first magnetic piece to the second magnetic piece. The keycap top may be textured, have a distinctive shape, or have a distinctive coloring, or have other features, to provide physical and visual cues for a user. The present disclosure also teaches a customizable keyboard and a method of customizing such keyboard using said interchangeable keycap tops.

The invention discloses a mirror image detection method based on generalized implicit reflection perception. A generalized implicit reflection perception framework based on visual prompt is constructed. Specifically, a prompt chain generation module is introduced, mirror image area features are sensed through thinking chain reasoning, and mirror image area feature sensing can be achieved even under the condition that no entity exists. The clear visual prompt is fixed or slightly updated at the image level, and the invention provides a prompt updating module based on a gating mechanism so as to realize feature level and semantic perception updating, reduce reflection interference and enhance a prompt chain. In addition, the invention proposes a cue injection module to drive the model to position the mirror. The modules provided by the invention are all plug-and-play. According to the method, the most advanced performance is realized with extremely low calculation complexity on four mirror image references, and the method exceeds single-mode, multi-mode and video-level methods of complete supervision and weak supervision. Good performance is also obtained on seven pieces of glass, camouflage and underwater benchmarks.

The invention discloses a multi-modal face living body detection method and device based on text enhancement, and the method comprises the steps: inputting a multi-modal face image into an image block embedding module and a visual prompt generation module, and carrying out the encoding through an image encoder, thereby obtaining an image feature and a visual prompt; inputting the dichotomy text description into a text embedding module, randomly initializing a text prompt vector, and encoding through a text encoder to obtain text features and text prompts; respectively enhancing image features and visual prompts by using a hybrid expert module and a bypass prompt enhancement module; performing information exchange on the visual prompt and the text prompt by using a text enhancement module to obtain an enhanced text prompt, and performing information exchange on the image feature and the text feature by using an image mask module to obtain an enhanced image feature; and finally calculating the similarity between the image features and the text features, and taking the category corresponding to the highest similarity as a detection result. According to the invention, the method can effectively enhance the discrimination capability of human face features, and improves the detection accuracy and generalization.

The invention relates to the technical field of intelligent visual detection, in particular to visual field detection equipment, system and method based on extended reality (XR), and the method comprises the steps that an image feature extraction module collects image data in real time, extracts features and eliminates optical distortion; the fixation deviation correction module analyzes eye movement data in real time, quantifies fixation deviation and triggers visual prompt and correction; the adaptive stimulation generation module dynamically encrypts to generate a test point image and adaptively adjusts the test point image according to visual field analysis and test requirements; the dynamic adjustment image module predicts and optimizes a test point presentation sequence according to image features and fixation points, automatically encrypts a high-probability defect area and quickly jumps to a key screening area; and the visual field defect identification module extracts image features, calculates the defect probability of each region in real time by using a Bayesian network in combination with subject response data, and locates a high-risk region. The system provides an efficient and accurate visual field detection scheme through the synergistic effect of multiple modules.

The invention belongs to the technical field of intelligent sports and computer vision crossing. The adaptive training method based on Tai Chi action recognition is provided, and space action features are extracted according to three-dimensional human skeleton key point coordinates of a human body; extracting multi-scale motion features of the human body in a time domain according to the spatial motion features; global attention aggregation is carried out according to the multi-scale motion features of the human body in the time domain, and a human body motion recognition result is obtained; according to a human body action recognition result, matching a corresponding Tai Chi style drawing theory; superposing multi-dimensional visual prompts into a Tai Chi motion video according to the Tai Chi motion style theory so as to intuitively prompt a user about action key points; calculating an action problem based on priori knowledge or a preset template, prompting an error position and an error reason by adopting graphs and / or characters, and giving an adjustment suggestion; according to the invention, more personalized training experience can be provided, cognitive load can be reduced, and training efficiency, training effect and cognitive learning fluency can be improved.

The invention discloses a visual and auditory concentration judgment method suitable for a driver of a vehicle, a host and a driver monitoring system. The method comprises the following steps: judging the visual concentration degree of the driver according to an image of the head of the driver obtained by a camera device in the vehicle; judging the cognitive concentration degree of the driver according to the sound in the vehicle obtained by a sound receiving device in the vehicle; according to the visual concentration degree and the cognitive concentration degree, whether a prompt is given to the driver or not is judged; and executing one or any combination of the following steps when the driver is judged to be prompted: enabling a display device in the vehicle to give out a visual prompt; and making a loudspeaking device in the carrier give out an auditory prompt.

The embodiment of the invention provides a remote sensing target segmentation method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-segmented remote sensing image and a sample library; calculating the feature similarity between the to-be-segmented target semantic tag and each sample, and screening out a target sample of which the feature similarity meets a preset requirement from the sample library; generating a target reference mask based on the target sample; and converting the target reference mask into a visual prompt, and inputting the visual prompt and the to-be-segmented target semantic tag into a segmentation model to obtain a segmentation result. Through combination of sample library assistance and visual prompt guidance, accurate segmentation of a specific target in a remote sensing image is realized, and dependence on dense manual annotation is eliminated. Meanwhile, the generalization ability of the model to the same kind of targets in different scenes is enhanced through double constraints of semantic tags and visual features.

The invention relates to a video action recognition model training method, a video action recognition method and a video action recognition device. The method comprises the following steps: acquiring a sample video frame image and an action category description text corresponding to the sample video frame image, inputting the sample video frame image and the action category description text into a to-be-trained recognition model, and recognizing the action category of the to-be-trained recognition model by an image encoder in the recognition model according to a preset visual prompt vector and the sample video frame image, generating a video embedding corresponding to a sample video frame image, generating a text embedding corresponding to an action category description text by a text encoder in the recognition model based on a preset text prompt vector and the action category description text, and constructing bidirectional comparison loss by taking the video embedding and the text embedding as positive sample pairs, and updating the visual prompt vector and the text prompt vector based on the bidirectional contrast loss to obtain a trained recognition model. By adopting the method, the video action recognition accuracy can be improved.

The present technology provides a three-dimensional video conference experience using readily available cameras and two-dimensional video displays by inserting the two-dimensional video of a remote video conference participant into a three-dimensional environment, and by using visual cues to create the perception of depth for the two-dimensional video of a remote video conference participant.

The invention discloses a random mode missing-oriented medical image unified diffusion generation method and a related device. The generation method comprises the following steps: acquiring one or more mode missing medical images and corresponding mode acquisition codes; sending the medical image and the modal acquisition code into a pre-trained visual prompt diffusion generation model to obtain a missing modal medical image; the pre-trained visual cue diffusion generation model comprises a visual cue extraction sub-network and a residual error enhanced U-Net network, and the visual cue extraction sub-network is used for extracting visual cues from the medical image according to modal acquisition codes; and the residual-enhanced U-Net network is used for continuously iteratively denoising the collected random Gaussian noise based on visual cues and target modal coding until a missing modal medical image is generated. The method focuses on exploring and improving the self-adaptive generation capability of the medical image of the diffusion generation model, and adopts the visual prompt extraction sub-network and the residual error enhanced U-Net network to realize unified and accurate generation of the medical image under the condition of random mode deficiency.