484 results about "Visual cueing" patented technology

The invention discloses a cross-domain face anti-counterfeiting detection method and device based on multi-modal text enhancement, and relates to the technical field of network information security, and the method comprises the steps: inputting two types of description texts into a pre-trained text encoder, and extracting text type features representing reality / deception, inputting the image into a pre-trained visual encoder to extract visual features; a trainable text prompt is added to each layer of the text encoder, a trainable visual prompt is added to each layer of the visual encoder, and the visual prompt of each layer of the visual encoder is obtained by converting the text prompt of the current layer through a full connection layer; and embedding the PFT module and the TIM module into a middle layer of each layer of a text encoder and a visual encoder to realize feature interaction fusion, obtaining cosine similarity and masks between text category features and visual features, and performing face true and false classification. Based on the PFT module and the TIM module, modal feature interaction is completed in the feature extraction process at the same time, and the cross-domain detection performance is improved.

The invention discloses a smart home scene understanding and interaction method and system based on multi-modal fusion, and the method comprises the steps: converting collected voice data into text data, carrying out the content understanding of collected image / video data through a multi-modal visual large model in order to more effectively understand the behaviors and potential intentions of a user, and carrying out the interaction of the collected image / video data. Then performing information extraction and reasoning on text data converted from voice data and contents understood by the multi-modal visual large model by utilizing the strong logical reasoning ability of the large language model; then, an instruction is generated based on the content of user interaction, visual cue words are generated from the instruction, real-time segmentation of the image / video content is implemented according to the visual cue words by adopting a multi-modal visual large model, and the large language model and the multi-modal visual model are fused in the intelligent home scene user interaction process through the visual cue words; and finally, performing three-dimensional processing on the segmented image, and realizing deep fusion with an AR / MR scene based on real-time segmentation of the scene.

The invention discloses a visible light and infrared image fusion target tracking method based on visual cue learning, and relates to the field of extended target tracking. Comprising the steps of inputting video image data, converting an image through a block embedding module, extracting multi-scale information through scale perception attention, extracting each modal feature through a double-branch backbone network, performing specific modal visual prompt learning and inter-modal visual prompt interaction, and performing cross fusion of an adapter and a prediction head. According to the method, a more reliable and efficient target tracking system is realized by introducing a multi-scale information extraction mechanism, enhancing visual prompt learning and interaction capability and developing a more advanced multi-modal fusion strategy, and through the improvements, the performance of target tracking in a complex environment can be remarkably improved, and the requirements of practical application are met.

A multi-material golf club head wherein the junction between the two different components made from different materials are seamlessly incorporated within existing visual cues of a golf club head is disclosed. More specifically, the present invention adjusts the design of the bond between the two different components to reflect the needs of the golf club head at the various visual cues around the perimeter of the golf club head.

The invention discloses a few-sample image quality evaluation method and device based on a visual language model, and relates to the technical field of image quality evaluation. Gradient adjustment meta-prompt image quality evaluation (GRMP-IQA) is introduced, meta-knowledge from various image distortions is merged by adopting a double-layer gradient optimization process through meta-prompt pre-training, so that text refining and initialization of visual prompt are realized, the adaptive capacity of a CLIP model to an IQA task is enhanced, and the accuracy of the image quality evaluation is improved. The sensitivity of prompt initialization to various IQA scenes is reduced; and then correcting the deviation introduced by excessive attention of semantic content of a visual-language model (CLIP) through quality perception gradient regularization, and ensuring that the image quality is evaluated more accurately, so that the pre-trained CLIP model adapts to a BIQA task with a small number of training samples.

The invention provides a zero sample anomaly detection method and device and electronic equipment, and relates to the technical field of image anomaly detection.The method comprises the steps that a to-be-detected image is acquired, and visual features are extracted through a CLIP model; performing visual enhancement on the visual features to obtain enhanced visual features; injecting the enhanced visual features into a learnable text prompt template to generate an adaptive text prompt; injecting the adaptive text prompt into a text encoder for encoding to obtain a text embedding representation; mapping the adaptive text prompt to a visual space to obtain a visual prompt, and inputting the visual prompt into the local visual features to obtain scale visual features; and based on the text embedding representation and the scale visual features, separately calculating an anomaly score and an anomaly positioning map of a preset scale, and fusing to obtain a detection result. According to the zero sample anomaly detection method and device and the electronic equipment provided by the invention, the generalization ability of the whole detection process is effectively improved, and the actual application requirements are further met.

The invention relates to a steel bridge disease detection and identification method based on a large language model, and belongs to the technical field of artificial intelligence and civil engineering crossing. According to the method, a cross-modal feature alignment mechanism is constructed through a pre-trained multi-modal large language model by fusing a steel bridge image and a field customized text prompt, and a cascade detection process of'component identification-disease classification-region segmentation 'is realized. Comprising the following steps: designing a structured text prompt word bank to enhance semantic consistency, and dynamically fusing general knowledge and instance features in combination with a mixed prompt mechanism; a multi-level cross-modal alignment strategy is adopted to generate an anomaly graph, and a disease area is accurately positioned; a visual prompt enhancement module is introduced to improve the multi-scale feature discrimination ability, and the robustness in a complex environment is adjusted and optimized through data self-adaption. Under the condition of few samples or even zero samples, high-sensitivity detection and pixel-level segmentation of steel bridge cracks, corrosion and other diseases are achieved, and the problems that a traditional method is low in efficiency, poor in generalization, high in labor cost and the like are effectively solved.

The embodiment of the invention provides a visual language model training method, an image tag prediction method and electronic equipment. According to the embodiment of the invention, a second visual language model is generated by training a first visual language model by using corresponding coding parameters in an image encoder and a text encoder in a fixed first visual language model and based on initial values and text samples of visual prompts respectively corresponding to various image classification tags; visual prompts respectively corresponding to various image classification labels are mined and learned, and the visual representation capability of a visual language model is enhanced. Further training a target visual language model by training a second visual language model based on learned visual prompts, text prompts and text samples, adjusting adapter parameters of model adapters configured for an image encoder and a text encoder respectively, and performing collaborative optimization on image feature extraction and text feature extraction; the process enables visual knowledge of the visual cue to be transferred to the text cue.

Redirected walking methods are described that may be implemented in virtual reality (VR) systems for guiding or redirecting users' movements within constrained physical environments or in unconstrained physical environments, while the users perceive that they are freely moving about within virtual worlds provided by the VR systems. The methods may involve rendering views of a VR world based at least in part on a user's movements in the real environment, adjusting the views according to the user's position and movements with respect to objects in and boundaries of the real environment, and displaying the adjusted views to the user. In response to visual cues introduced by the methods, the user may adjust their path in the real world environment. For example, a rotation of the VR environment introduced by a shift of one or more views may cause the user to turn in a direction opposite to the rotation.

The embodiment of the invention provides a sample set construction method, a question and answer model training method, a question and answer processing method, a request processing method and a task platform, and the sample set construction method comprises the steps: inputting a sample image into an image annotation model, and obtaining a global annotation result of the sample image; identifying a target object in the sample image, marking the target object in the sample image based on an obtained identification result to obtain a global view, and intercepting the target object to obtain a local view; the global view, the local view and visual prompt information are input into the image annotation model, a local annotation result of the sample image is obtained, and the visual prompt information is used for prompting the image annotation model to execute an annotation task on the global view based on the local view; and constructing a sample set based on the sample image, the global labeling result and the local labeling result. According to the method, the accuracy and the comprehensiveness of the annotation result of the image can be improved.

The invention provides a method and system for efficiently marking and segmenting a focus area in a medical image, and the method comprises the steps: S1, collecting a medical image data set, and dividing the medical image data set into a training set and a test set; s2, selecting a reference image from the training set, carrying out initial labeling, generating a visual prompt, and obtaining the reference image with the visual prompt; s3, constructing a deep convolutional neural network architecture, and obtaining a trained model through model training; s4, performing prediction segmentation on the unlabeled image in the test set by using the trained model to generate a prediction segmentation result; and S5, checking and correcting the prediction segmentation result, and improving the model performance through interactive optimization. According to the method, a doctor only needs to carry out initial labeling on a small number of focus areas, the problem that an existing fully-supervised segmentation method seriously depends on a large amount of high-quality labeling data is solved, the requirement for the high-quality labeling data is remarkably reduced, and the workload of medical staff is relieved.

The invention discloses a product defect unsupervised semantic segmentation method and device based on a pre-training model, and is used for solving the technical problem that the defect classification precision is not ideal due to an existing industrial defect unsupervised semantic segmentation method. The method comprises the following steps: acquiring a to-be-detected industrial product image, an industrial product reference image corresponding to the to-be-detected industrial product image and a plurality of to-be-matched defect categories; a preset segmentation network of a self-learning point generation prompter based on defect potential space features is adopted to generate a target defect segmentation mask according to a to-be-detected industrial product image and an industrial product reference image, and a target cutting image is generated based on the target defect segmentation mask, so that the target cutting image with visual prompt can be generated; and processing the target cutting image and a plurality of defect categories to be matched through a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module, and combining segmentation and classification, thereby improving the defect classification precision.

A computer system, method, and program product facilitate human-computer interaction. A processor set receives a non-text visual cue and natural language instruction regarding a scene. The processor set converts the non-text visual cue into a textual location information indicating a portion of an image representing the scene. A language machine learning model is triggered by using the textual location information, the natural language instruction, and the image representing the scene as input. The language machine learning model outputs a response to the input.

The invention provides an experiment task execution method and device.The method comprises the steps that task information of a target experiment task is divided based on a visual language model, a subtask sequence is obtained, and the subtask sequence is formed by arranging multiple subtasks from front to back according to the execution sequence; processing the text information corresponding to the current sub-task and the experiment image before the current sub-task is executed through the visual language model from the first sub-task in the sub-task sequence to obtain a visual prompt image, and executing the visual prompt image through a visual language action model. And processing based on the text information, the experiment image and the visual prompt image, guiding a robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, and determining that the target experiment task is completed. According to the method, the success rate, the operation safety and the regulation compliance of the experiment task are effectively improved, and the method has high universality and safety.

The invention discloses a remote sensing image change detection method based on fine tuning CLIP, and the method comprises the steps: obtaining an original dual-time remote sensing image, and carrying out the preprocessing of the remote sensing image; constructing an initial remote sensing image change detection model, and training the initial remote sensing image change detection model by using the training set to obtain a target remote sensing image change detection model; and performing change detection on the remote sensing image by using the target remote sensing image change detection model. According to the method, a large model CLIP pre-trained in a large-scale data set is introduced into a change detection task, and the method aims at guiding the model to pay attention to universal features of changing ground features, so that the generalization ability of the model is improved; according to the method, the CLIP is finely adjusted by adopting a visual prompt method, so that the CLIP can be well migrated in a remote sensing downstream task; in addition, when a loss function is constructed, Dice coefficient loss is introduced to enhance the sensitivity of the model to a positive example.

The method involves using a digital badge device worn on the chest of a user to visually convey personal attributes to others in line of sight. It begins by configuring the badge device to communicate wirelessly with the user's mobile device. Next, a personal attribute of the user is determined based on input from the user or device-captured data. This attribute could relate to the user's affiliation, emotional state, physical activity, and more. Finally, a pre-defined color and / or symbol code representing the determined attribute is displayed on the badge device's screen for others to see. This provides an easily interpretable visual cue to communicate specific personal attributes to different users. By employing this method, the badge wearer can convey information about themselves without the need for verbal communication. The method offers a convenient and efficient way to signal personal attributes in situations where direct interaction may be limited or impractical.

The invention provides a remote sensing image target statistical method and system fusing a large language model and visual prompt driving. The method comprises the following steps: acquiring a remote sensing instance segmentation image to be processed and a visual prompt thereof; inputting a to-be-processed remote sensing instance segmentation image and a visual prompt thereof into the trained remote sensing image target statistical model, and outputting a remote sensing image target statistical result; the training comprises the following steps: introducing a large language model and visual cue into an encoder architecture of a GrondingDINO model to obtain a remote sensing image target statistical model; inputting a remote sensing instance segmented image and the visual cue thereof into an encoder, and outputting an image feature, a visual cue feature and a text feature; the feature intensifier carries out fusion processing on the output of the encoder; a language-guided query selection module calculates cross-modal query according to the fusion processing result, and a cross-modal decoder obtains a target statistical result of the image based on the fusion processing result and the cross-modal query; and training by using the training data and outputting the trained model.

The invention discloses a video understanding method and device based on a multi-modal large model and a medium, and the method comprises the steps: carrying out the frame-by-frame fusion of visual prompt information corresponding to a predetermined first frame and a video frame sequence through a dynamic Alpha mixing technology; extracting multi-frame features from the fused video frame sequence, and integrating the multi-frame features to generate corresponding space-time visual representation; for each current frame in the video frame sequence, fusing the visual prompt information of the previous frame corresponding to the current frame with the visual features of the current frame, and generating visual prompt information of all frames after the first frame in the video frame sequence; and according to the dynamic Alpha mixed coefficient, performing smooth transition on the visual prompt information of every two adjacent frames in the video frame sequence so as to generate a corresponding video understanding result through an autoregression language model in combination with the text instruction and the space-time visual representation.

Implementations of a visual cue system for conveying road state data to one or more vehicles traveling on a roadway can include a set of visual cue units that receive road state data and selectively issue a visual cue based on the road state data, at least one visual cue unit including a data processing unit configured to process the road state data and determine a light pattern based on a set of light pattern generation rules, and a light pattern display that selectively issues at least a portion of the light pattern to the one or more vehicles traveling on the roadway.

The invention discloses a visual prompt guided remote sensing image similar target detection method and device. The method comprises the following steps: selecting and preprocessing remote sensing data, and performing data enhancement on images and visual prompts to obtain a preprocessed data set; constructing a target detection model; using a CNN backbone network to perform feature extraction on the preprocessed image, and inputting the extracted image into an encoder of a Transform; carrying out feature extraction on the visual prompt by using a CNN backbone network, adding the extracted visual prompt and a query object, and inputting the added visual prompt and the query object into a Transform decoder; the preprocessed image and the processing data of the visual prompt are fused in a decoder, a bounding box and a label are output through an FNN, and a prediction result is obtained; and training the network by using the loss function and the prediction result to obtain a final target detection model. According to the method, a multi-classification task is converted into a dichotomy task (matched or mismatched), an adopted remote sensing data set is processed, and Visual Prompt is introduced, so that a DETR model is expanded to a similar target detection task, and the model performance is improved.

An autonomous landing system includes a vertical landing aircraft, a visual cue coupled to an offshore landing platform and including a visual indicator located that extends vertically across a horizon and is maintained parallel with the horizon, and an autonomous control system configured to control a position of the aircraft with respect to the visual indicator to maneuver the aircraft into a hovering, predefined landing position above the offshore landing platform, and control the position of the aircraft with respect to the visual indicator to cause the aircraft to descend from the landing position and land on the offshore landing platform.

The invention discloses an unsupervised continuous anomaly detection method and system for multi-mode prompt memory, and relates to the technical field of computer vision and industrial image detection. The method comprises the following steps: acquiring industrial images to form a data set; an unsupervised anomaly detection model is constructed, the unsupervised anomaly detection model comprises a visual branch and a text branch which are respectively used for extracting visual features and text features, learnable visual prompts and learnable text prompts are introduced into the unsupervised anomaly detection model, and then the visual features and the text features are fused by using a self-adaptive fusion mechanism to obtain an anomaly detection result; training learnable visual prompts and learnable text prompts in the unsupervised anomaly detection model by using the data set; and performing anomaly detection on the industrial data by using the trained unsupervised anomaly detection model. According to the method, multi-modal information can be fused in the learning process, the continuous learning ability is achieved, and efficient unsupervised anomaly detection of industrial products is achieved.

The invention provides a carsickness relieving method and system based on display screen dynamic background image enhancement, and the method comprises the steps: collecting the acceleration and posture data of a vehicle through a sensor, and carrying out the real-time fusion processing of the data through a data fusion algorithm; the input display picture is cut and zoomed, a black and white stripe area is set, black and white stripes are controlled to move, the rotation angle of an effective display area is adjusted in real time through an algorithm by means of sensor data, black spots are set in the effective display area to move up and down for visual compensation and other processing, and video signals are transcoded and processed; and the video content is output to an adaptive display interface, so that the display of the video content is realized, carsickness is relieved by introducing visual prompts to passengers in the vehicle, and the comfort and the sense of safety of the passengers are improved.

The invention discloses a passenger car component fault detection method and system based on visual prompt and a visual large model, and the method comprises the steps: carrying out the position marking of a target in each sample image for each type of component sample image set, splicing a panoramic image, and generating a mask image; describing the exception type of each type of part and the details of each exception type through the exception description text; extracting multi-scale image features of the panoramic image and the mask image, and fusing the multi-scale image features to obtain visual prompt features; extracting a multi-scale image feature of the to-be-detected part image, fusing the multi-scale image feature with the visual prompt feature to obtain a multi-scale image feature of the to-be-detected part, inputting the multi-scale image feature into the part positioning model, and positioning a target part image; and fusing the visual features of the target part image with the text features of the corresponding abnormal description text by using a visual large model, and predicting and outputting a fault detection result. According to the invention, precise part positioning and fault detection can be realized.

One or more light sources may be integrated with a medication delivery device, and states of the light sources may be changed. For example, a first lighting state change may be performed, such as based on a detection that an applied force meets or exceeds a threshold break force or an interaction of a movable component with a proximal sensor integrated with the medication delivery device. The first lighting state change may provide visual feedback indicating that an injection has begun. Additionally, a second lighting state change may be performed, such as based on a detection that a resistive force meets or exceeds a threshold back force or an interaction of the movable component with a distal sensor integrated with the medication delivery device. The second lighting state change may indicate that a complete dosage of the medication has been delivered from the medication delivery device.

The invention provides a method and a device for selectively positioning a sound source based on visual prompt, a medium and a product. The method comprises the following steps: acquiring a mixed audio signal and a prompt image; the mixed audio signal comprises audio signals corresponding to different sound events triggered by at least two different sound sources; the prompt image is associated with a target sound source, and the target sound source is a required sound source selected and positioned from different sound sources; the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event; processing the mixed audio signal and the prompt image through a preset cross-instance audio-visual positioning model, including estimating a target mask; and outputting a target arrival direction corresponding to the target sound source based on the target mask. According to the invention, the specific sound source corresponding to the specific sound event is selected and positioned from a plurality of different sound events triggered by a plurality of sound sources, and the application range is wide.

The invention relates to the technical field of medical image processing, in particular to an ultrasonic image segmentation method based on edge guidance, and the method comprises the steps: inputting an ultrasonic image into an edge extraction branch and an image encoder; outputting an edge mask image corresponding to the image through the edge extraction branch; performing noise suppression and topological correction on the edge mask image to generate a closed edge constrained by an anatomical structure; generating prompt box information based on the closed edge, and inputting the prompt box information to a prompt encoder; fusing the image prompt features output by the prompt encoder, the image features extracted by the image encoder and the edge mask information; the fusion features are input into a decoding module, a final segmentation result is obtained, the synergistic effect of edge information and visual prompt is fully utilized, the perception ability of the model for the anatomical structure in the ultrasonic image is improved, and the accuracy and robustness of segmentation are remarkably improved. Therefore, the problems of fuzzy boundary, inaccurate prompt, weak structure identification capability and the like in related technologies are solved.

Implementations relate to dynamically adapting a given assistant output based on a given persona, from among a plurality of disparate personas, assigned to an automated assistant. In some implementations, the given assistant output can be generated and subsequently adapted based on the given persona assigned to the automated assistant. In other implementations, the given assistant output can be generated specific to the given persona and without having to subsequently adapt the given assistant output to the given persona. Notably, the given assistant output can include a stream of textual content to be synthesized for audible presentation to the user, and a stream of visual cues utilized in controlling a display of a client device and / or in controlling a visualized representation of the automated assistant. Various implementations utilize large language models (LLMs), or output previously generated utilizing LLMs, to reflect the given persona in the given assistant output.

Embodiments of the present disclosure provide a method, device, and medium for training a large scale object foundation model. The method comprises obtaining a training dataset comprising a plurality of subsets for a plurality of object perception tasks, wherein a sample in the training dataset comprises an image with an object, a prompt indicating the object, and labeled object perception information of the image. The method further comprises generating, by the image encoder, an image feature based on the image. The method further comprises generating, by the text encoder or the visual prompt encoder, a prompt embedding based on the prompt. The method further comprises generating, by the object decoder, object perception information of the object based on the image feature and the prompt embedding. In addition, the method further comprises training the object processing model based on the generated object perception information and the labeled object perception information.

The present disclosure teaches a keycap with an interchangeable keycap top. The keycap comprises a keycap base and a keycap top. Wherein, a first magnetic piece connects to the keycap base, and a second magnetic piece connects to the keycap top. Both magnetic pieces can be a magnet or of a magnetic material, and at least one of the magnetic pieces is a magnet. The keycap top can thus be secured onto the keycap base by attaching the first magnetic piece to the second magnetic piece. The keycap top may be textured, have a distinctive shape, or have a distinctive coloring, or have other features, to provide physical and visual cues for a user. The present disclosure also teaches a customizable keyboard and a method of customizing such keyboard using said interchangeable keycap tops.