Visual task processing method, visual processing model training method and image generation method

Abstract

Provided in the embodiments of the present description are a visual task processing method, a visual processing model training method and an image generation method. The visual task processing method comprises: acquiring an image to be processed and prompt text; inputting said image into a visual encoding layer of a visual processing model for encoding, so as to obtain visual input features, and inputting the prompt text into a text encoding layer of the visual processing model for encoding, so as to obtain text input features; inputting the visual input features and the text input features into a feature processing layer, and executing a target visual task, so as to obtain multi-modal output features; and inputting the multi-modal output features into a decoding layer, and on the basis of a plurality of text feature units, positioning a corresponding target image region in said image, so as to obtain a target image in which the target image region is highlighted. No additional task decoder is added for a model, a variety of visual tasks for fine-grained perception are completed, the complexity of the architecture of the model is not increased, the scalability of the model is retained, and the synergistic effect between different visual tasks is also promoted.

Description

Visual task processing, training of visual processing models, and image generation methods

[0001] This disclosure claims priority to Chinese Patent Application No. 202510199044.X, filed on February 21, 2025 with the China National Intellectual Property Administration, entitled “Visual Task Processing, Training of Visual Processing Model and Image Generation Method”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the fields of neural networks, vision technology, and image processing, and particularly to a method for visual task processing, training of visual processing models, and image generation. Background Technology

[0003] With the development of neural network technology, visual processing models have made significant progress in image-level content understanding.

[0004] Currently, mainstream visual processing models primarily understand image-level content but lack the ability to analyze finer-grained entities or attributes within images. This can lead to the illusion that the output content does not match the image content. To address this, a corresponding decoder is added for each fine-grained visual task. This decoder decodes the features of a unified output to obtain the corresponding output results for each visual task.

[0005] However, such fine-grained perception-based visual processing models significantly increase the complexity of the model architecture and the training process, hindering their application to more visual tasks. This not only requires additional computational resources but also limits the model's scalability and impedes the synergistic effects between multiple visual tasks. Therefore, there is an urgent need for a visual processing model that can implement multiple fine-grained perception tasks without adding an extra decoder, and a method to complete various fine-grained perception-based visual task processing tasks based on this model. Summary of the Invention

[0006] In view of the above, embodiments of this specification provide a visual task processing method. One or more embodiments of this specification also relate to a method for training a visual processing model, an image generation method, a visual task platform, a model training platform, a visual task processing device, a training device for a visual processing model, an image generation device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a visual task processing method is provided, comprising:

[0008] Acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed;

[0009] The image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features, and the prompt text is input into the text encoding layer of the visual processing model to obtain text input features. The visual processing model also includes a feature processing layer and a decoding layer.

[0010] Visual input features and text input features are input into the feature processing layer to perform the target visual task and obtain multimodal output features including multiple text feature units. Each text feature unit represents the classification result of the target visual task performed on the corresponding image region in the image to be processed.

[0011] The multimodal output features are input into the decoding layer, and the corresponding target image region in the image to be processed is located based on multiple text feature units to obtain the target image that highlights the target image region.

[0012] According to a second aspect of the embodiments of this specification, a method for training a visual processing model is provided, applied to a model training platform, comprising:

[0013] Acquire sample images, sample prompt text for performing the target visual task on the sample images, and label images, wherein the label image area is highlighted in the label image;

[0014] The sample image is input into the visual encoding layer of the initial visual processing model to obtain the sample visual input features, and the prompt text is input into the text encoding layer of the initial visual processing model to obtain the sample text input features. The initial visual processing model also includes a feature processing layer and a decoding layer.

[0015] The sample visual input features and sample text input features are input into the feature processing layer to perform the target visual task and obtain the predicted multimodal output features including multiple text feature units. Each text feature unit represents the predicted classification result of the target visual task performed on the corresponding image region in the sample image.

[0016] The predicted multimodal output features are input into the decoding layer, and the corresponding predicted image regions in the sample image are located based on multiple text feature units;

[0017] Based on the predicted image region and the labeled image region, the initial visual processing model is trained to obtain the trained visual processing model.

[0018] According to a third aspect of the embodiments of this specification, an image generation method is provided, applied to a vision task platform, comprising:

[0019] Obtain the first prompt text used to generate the target image that highlights the target image region;

[0020] Using a visual processing model, an image generation task is performed based on a first prompt text to generate an image to be processed and a second prompt text for performing a target visual task on the image to be processed. The target visual task is then performed based on the second prompt text on the image to be processed to obtain the target image. The steps of performing the target visual task implement the steps of the above-mentioned visual task processing method.

[0021] According to a fourth aspect of the embodiments of this specification, a visual task platform is provided, including a visual processing interface and a response unit;

[0022] The visual processing interface is used to acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed;

[0023] The response unit is used to execute the steps of the above-described visual task processing method.

[0024] According to a fifth aspect of the embodiments of this specification, a model training platform is provided, including a model training interface and a response unit;

[0025] The model training interface is used to acquire sample images, sample prompt text for performing target visual tasks on the sample images, and label images, wherein the label image area is highlighted in the label image;

[0026] The response unit is used to execute the steps of the training method for the aforementioned visual processing model.

[0027] According to a sixth aspect of the embodiments of this specification, a visual task processing apparatus is provided, comprising:

[0028] The first acquisition module is configured to acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed;

[0029] The first encoding module is configured to input the image to be processed into the visual encoding layer of the visual processing model, encode it to obtain visual input features, and input the prompt text into the text encoding layer of the visual processing model, encode it to obtain text input features. The visual processing model also includes a feature processing layer and a decoding layer.

[0030] The first processing module is configured to input visual input features and text input features into the feature processing layer, perform the target visual task, and obtain multimodal output features including multiple text feature units, wherein any text feature unit represents the classification result of the target visual task performed on the corresponding image region in the image to be processed;

[0031] The first decoding module is configured to input multimodal output features into the decoding layer, locate the corresponding target image region in the image to be processed based on multiple text feature units, and obtain the target image that highlights the target image region.

[0032] According to a seventh aspect of the embodiments of this specification, a training apparatus for a visual processing model is provided, applied to a model training platform, comprising:

[0033] The second acquisition module is configured to acquire a sample image, sample prompt text for performing a target visual task on the sample image, and a label image, wherein the label image contains a highlighted label image area.

[0034] The second encoding module is configured to input the sample image into the visual encoding layer of the initial visual processing model to encode the sample visual input features, and input the prompt text into the text encoding layer of the initial visual processing model to encode the sample text input features. The initial visual processing model also includes a feature processing layer and a decoding layer.

[0035] The second processing module is configured to input the sample visual input features and sample text input features into the feature processing layer, perform the target visual task, and obtain the predicted multimodal output features including multiple text feature units, wherein any text feature unit represents the predicted classification result of performing the target visual task on the corresponding image region in the sample image;

[0036] The second decoding module is configured to input the predicted multimodal output features into the decoding layer and locate the corresponding predicted image region in the sample image based on multiple text feature units;

[0037] The second training module is configured to train the initial visual processing model based on the predicted image region and the label image region, thereby obtaining the trained visual processing model.

[0038] According to an eighth aspect of the embodiments of this specification, an image generation apparatus is provided, applied to a vision task platform, comprising:

[0039] The third acquisition module is configured to acquire first prompt text for generating a target image that highlights the target image region;

[0040] The generation module is configured to use a visual processing model to perform an image generation task based on a first prompt text to generate an image to be processed and a second prompt text for performing a target visual task on the image to be processed. Based on the second prompt text, the module performs a target visual task on the image to be processed to obtain a target image. The step of performing the target visual task implements the steps of the above-described visual task processing method.

[0041] According to a ninth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0042] Memory and processor;

[0043] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned visual task processing method, visual processing model training method, or image generation method.

[0044] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described visual task processing method, visual processing model training method, or image generation method.

[0045] According to the eleventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual task processing method, visual processing model training method, or image generation method.

[0046] In one embodiment of this specification, the image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features, and the prompt text is input into the text encoding layer of the visual processing model to obtain text input features. The visual processing model further includes a feature processing layer and a decoding layer, ensuring accurate representation of image and text inputs within the model. The visual and text input features are input into the feature processing layer to perform the target visual task, obtaining multimodal output features including multiple text feature units. Each text feature unit corresponds to a different region in the image, representing the classification result of performing the target visual task on the image region. This completes the fine-grained parsing and classification of the image region, ensuring that each text feature unit accurately corresponds to a specific region in the image and performs a fine-grained target visual task on the image region. The multimodal output features are input into the decoding layer, locating the corresponding target image region in the image to be processed based on the multiple text feature units, obtaining a target image highlighting the target image region. This not only ensures accurate location and recognition of the target entity or attribute but also greatly improves the visualization effect of the target image, intuitively demonstrating the model's output structure. This method does not add an extra task decoder to the model, accomplishes a variety of fine-grained perception visual tasks, does not increase the complexity of the model architecture, preserves the scalability of the model, and also promotes synergy between different visual tasks. Attached Figure Description

[0047] Figure 1 is a schematic diagram comparing the architectures of various visual task processing methods;

[0048] Figure 2 is a schematic diagram of the architecture of a visual task processing method;

[0049] Figure 3 is a flowchart of a visual task processing method provided in one embodiment of this specification;

[0050] Figure 4 is a schematic diagram of the architecture of a visual task processing method provided in one embodiment of this specification;

[0051] Figure 5 is a flowchart of a training method for a visual processing model provided in one embodiment of this specification;

[0052] Figure 6 is a flowchart of an image generation method provided in one embodiment of this specification;

[0053] Figure 7 is a flowchart of a visual task processing method for a multimodal intelligent question answering scenario provided in an embodiment of this specification.

[0054] Figure 8 is a schematic diagram of the structure of a vision task platform provided in one embodiment of this specification;

[0055] Figure 9 is a schematic diagram of the structure of a model training platform provided in one embodiment of this specification;

[0056] Figure 10 is a schematic diagram of the structure of a visual task processing device provided in one embodiment of this specification;

[0057] Figure 11 is a schematic diagram of the structure of a training device for a visual processing model provided in one embodiment of this specification;

[0058] Figure 12 is a schematic diagram of an image generation apparatus provided in one embodiment of this specification;

[0059] Figure 13 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0060] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0061] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0062] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0063] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0064] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically including hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0065] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0066] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0067] Fine-grained visual perception: refers to fine-grained recognition of image content, such as locating entities or attributes, and detecting and segmenting entities or attributes of specific categories.

[0068] Transformer models: Neural network models that use self-attention to process sequential data. They are also widely used in the field of computer vision, such as ViT (Vision Transformer), for performing visual tasks.

[0069] Multimodal large language models: Multimodal large language models are a type of artificial intelligence model that integrates multiple information modality processing capabilities based on large language models. Compared with traditional multimodal models, they utilize the powerful reasoning capabilities of large language models and demonstrate strong potential and emergent capabilities in various visual tasks and multimodal understanding tasks.

[0070] Visual language tasks: generally refer to coarse-grained perception of image content, such as image description and image question answering tasks.

[0071] Currently, mainstream visual processing models primarily focus on understanding image-level content, such as image description and image question answering, but they cannot analyze finer-grained entities or attributes within images. This makes the model prone to producing the illusion that the output content does not match the image content. Inaccurate understanding of image content negatively impacts the user experience in practical applications. For fine-grained perception visual processing models, adding a corresponding decoder for each fine-grained perception visual task and decoding the features of a unified output to obtain the corresponding output results for each visual task significantly increases the complexity of the model architecture and training process. This hinders the expansion to more visual tasks, requires additional computing resources, limits model scalability, and impedes the synergistic effects between multiple visual tasks. Furthermore, the performance requirements for multi-task training far exceed those for single-task training.

[0072] Specifically, Figure 1 shows a comparative architectural diagram of various visual task processing methods, as shown in Figure 1:

[0073] Task-based decoder-based solutions require adding a decoder for each fine-grained perception task, which significantly increases the complexity of the model architecture and makes it difficult to extend to more tasks.

[0074] Text-based methods often only represent bounding boxes, and when representing segmentation maps, they can only use imprecise polygons or very inefficient text categories, resulting in poor segmentation performance.

[0075] This specification presents an alternative approach that eliminates the need for a task-specific decoder, thus avoiding increased model complexity and training difficulty. The unified modeling approach also promotes synergy between tasks, enabling different tasks to assist each other, and allowing the model to perform more complex tasks by combining multiple tasks.

[0076] Figure 2 shows a scheme based on a task decoder. Figure 2 illustrates the architecture of a visual task processing method:

[0077] The visual processing model is trained to output labels such as [DET] and [SEG]. The corresponding label features are then input to an additional detection box decoder to support detection, or a mask decoder to support segmentation.

[0078] This specification provides a visual task processing method, and also relates to a visual processing model training method, an image generation method, a visual task platform, a model training platform, a visual task processing device, a visual processing model training device, an image generation device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0079] Referring to Figure 3, which shows a flowchart of a vision task processing method according to an embodiment of this specification, the method includes the following specific steps:

[0080] Step 302: Obtain the image to be processed and the prompt text for performing the target visual task on the image to be processed.

[0081] The image to be processed is the raw image data that needs to be processed by a visual task. The image to be processed may include at least one entity or at least one attribute. The visual task needs to be performed to locate, classify, detect, or segment at least one entity or at least one attribute in the image to be processed. The image to be processed has a corresponding resolution and includes multiple image regions, each of which includes at least one pixel.

[0082] Visual tasks are visual analysis tasks that perform fine-grained visual perception of images, and are used to extract specific types of information from images or to perform specific operations. These include, but are not limited to, visual detection tasks, visual segmentation tasks, entity classification tasks, key point detection tasks, attribute classification tasks, etc.

[0083] The prompt text is text data used to guide the visual processing model in performing visual tasks. It guides the model on how to perform visual tasks and provides additional contextual information for processing images.

[0084] The process of obtaining the image to be processed and the prompt text for performing the target visual task on the image to be processed can be implemented in several ways. One option is to receive the image to be processed and the prompt text for performing the target visual task on the image to be processed sent by the front end. Another option is to obtain the image to be processed and the prompt text for performing the target visual task on the image to be processed from a predefined dataset. Yet another option is to obtain the image to be processed and the prompt text for performing the target visual task on the image to be processed through a data interface. No further limitations are specified here.

[0085] For example, on a task platform that has deployed a multimodal large language model pre-learned for various visual tasks, a landscape photo x_v sent by a user through the front end is received, along with a prompt text x_t for performing a visual segmentation task on the landscape photo: "Please segment out the sheep in the image."

[0086] It acquires the image to be processed and the prompt text for performing the target visual task on the image to be processed, providing multimodal data support of images and text for the subsequent execution of the target visual task in the model.

[0087] Step 304: Input the image to be processed into the visual encoding layer of the visual processing model to obtain visual input features, and input the prompt text into the text encoding layer of the visual processing model to obtain text input features. The visual processing model also includes a feature processing layer and a decoding layer.

[0088] Visual processing models are neural network models that perform visual tasks on images guided by prompting text, typically trained using deep learning techniques. A visual processing model includes a visual encoding layer, a text encoding layer, a feature processing layer, and a decoding layer, but does not include additional decoding layers. Visual processing models include, but are not limited to, Transformer models and multimodal large language models.

[0089] The visual encoding layer is the part of the model that encodes raw image data into feature vectors that the model can process. Through word segmentation, embedding, convolution operations, and other methods, the visual encoding layer extracts key visual features from the raw image data. For example, a visual word segmenter.

[0090] The text encoding layer is the part of the model that encodes text data into feature vectors that the model can process. Through methods such as word segmentation, embedding, and convolution, the text encoding layer extracts key text features from the text data. For example, a text segmenter.

[0091] Visual input features are feature encoding vectors extracted from the image to be processed to describe the image content. They are usually high-dimensional encoded vectors that can capture visual information such as shape, color, and texture in the image to be processed.

[0092] Text input features are feature encoding vectors extracted from the prompt text to describe the task content of the target visual task. They are usually high-dimensional encoded vectors that can capture textual information such as semantic information, keywords and their contextual relationships.

[0093] For example, a landscape photo x_v is input into the visual tokenizer of the multimodal large language model for visual encoding to obtain the visual input feature 〖Token〗_v, and the prompt text x_t is input into the text tokenizer of the multimodal large language model for text encoding to obtain the text input feature 〖Token〗_t.

[0094] The image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features, and the prompt text is input into the text encoding layer of the visual processing model to obtain text input features. The visual processing model also includes a feature processing layer and a decoding layer to ensure that the image input and text input are accurately represented in the model.

[0095] Step 306: Input the visual input features and text input features into the feature processing layer, perform the target visual task, and obtain multimodal output features including multiple text feature units, wherein any text feature unit represents the classification result of the target visual task performed on the corresponding image region in the image to be processed.

[0096] The feature processing layer is the model part that performs visual tasks on feature vectors. By combining visual and textual information, the feature processing layer uses attention mechanisms, feature fusion (such as addition, multiplication, and bilinear pooling), transformations, and mappings (such as fully connected layers or other forms of nonlinear transformations to adjust feature dimensions or distribution) to perform visual tasks, obtaining multimodal output features that include multiple text feature units. For example, the Transformer model for multimodal vision tasks.

[0097] Multimodal output features are multimodal feature encoding vectors of the task results of the target visual task output by the visual processing model, including but not limited to at least one of the following visual modal output features and textual modal output features. By combining visual and textual information, multimodal output features utilize attention mechanisms, feature fusion (such as addition, multiplication, and bilinear pooling), transformations, and mappings (such as fully connected layers or other forms of nonlinear transformations) to achieve a deep understanding and parsing of image content and execute the task results of the target visual task. These results include multiple textual feature units representing the classification results of the corresponding image regions in the image to be processed when performing the target visual task.

[0098] Text feature units are the basic units used to represent the classification results of a target visual task performed on a corresponding image region in the image to be processed. Each text feature unit is associated with a specific region in the image to be processed and carries information about the task performance result for that region. For example, in a visual detection task, it represents the coordinates of a detection box; in a visual segmentation task, it represents the label of the segmentation map (e.g., ...). <mask>These units enable the model to perform fine-grained tasks for different parts of an image and ensure the accuracy of the results. For example, in a visual segmentation task, the prompt text is "Mark all grass". The text feature unit represents the classification of each pixel or group of pixels in the image, determining whether it belongs to grass, and using the corresponding label (e.g., ...). <mask>These regions are labeled, and the multimodal output features will include text feature units of all identified entity regions that may belong to the grassland. Finally, the specific location and bounding box of the grassland are decoded and output.

[0099] Optionally, before step 306, the following specific features are also included: splicing visual input features and text input features to obtain multimodal input features;

[0100] Correspondingly, step 306 includes the following specific steps: inputting the multimodal input features into the feature processing layer, performing the target vision task, and obtaining multimodal output features including multiple text feature units.

[0101] For example, visual input feature 〖Token〗_v and text input feature 〖Token〗_t are concatenated to obtain multimodal input feature Token. The multimodal input feature Token is then input into the feature processing layer to perform a visual segmentation task, obtaining visual output feature h_v and text output feature h_t: This is <mask>The text output features include multiple text feature units (tokens), which represent mask labels for performing visual segmentation tasks on corresponding image regions in landscape photos. <mask>This is used to identify whether the image region belongs to the "sheep" category.

[0102] Visual input features and text input features are input into the feature processing layer to perform the target visual task and obtain multimodal output features including multiple text feature units. The text feature units correspond to different regions in the image and represent the classification results of the target visual task performed on the image region. This completes the fine-grained parsing and classification of the image region, so that the text feature units can accurately correspond to specific regions in the image and perform fine-grained target visual tasks on the image region.

[0103] Step 308: Input the multimodal output features into the decoding layer, locate the corresponding target image region in the image to be processed based on multiple text feature units, and obtain the target image that highlights the target image region.

[0104] The decoding layer is the part of the model that decodes the feature vectors that the model can process into output image data, such as bounding boxes in visual detection tasks and segmentation maps in visual segmentation tasks. The decoding layer performs autoregressive prediction of image regions in the data to be processed based on multimodal output features. Compared to existing visual processing models, the decoding layer maintains a consistent structure without change; only the parameters have been trained and adjusted for various visual tasks.

[0105] The target image region in the image to be processed is the image region where the target visual task is performed, and the determined task result is located. The target image region can be any part of the image with specific attributes or satisfying specific conditions. For example, the target image region includes a specific entity (such as an animal or vehicle) or a region with a specific attribute (such as color or shape). Based on this, visual detection and visual segmentation tasks are completed. For example, the visual processing model will identify entity regions in the image to be processed that may belong to grass and add classification results to these image regions. Based on these classification results, feature retrieval is performed to add mask labels to all image regions, obtaining a mask image. Image regions with mask labels are rendered in one style, and image regions without mask labels are rendered in another style. The mask labels are rendered in reverse order.

[0106] The target image is the image data used for the visual task. It includes the analysis results of specific entities or attributes in the original image to be processed. These results are typically highlighted visually, such as by overlaying detection boxes, using different styles of distinguishable rendering, or other forms of annotation to highlight the target image region. The target image has a corresponding resolution and includes multiple image regions, each containing at least one pixel. Generally, the target image and the image to be processed are resolution-aligned.

[0107] The multimodal output features are input into the decoding layer, and the corresponding target image region in the image to be processed is located based on multiple text feature units to obtain the target image that highlights the target image region. One possible approach is to input the multimodal output features into the decoding layer, locate the corresponding target image region in the image to be processed based on multiple text feature units, and render to obtain the target image that highlights the target image region. The rendering methods include, but are not limited to, rendering detection boxes and different style differentiation rendering.

[0108] For example, visual output features h_v and text output features h_t are input into the decoding layer, based on mask labels of multiple text feature unit tokens. <mask>Locate the target image region corresponding to the "sheep" target segmentation entity in the landscape photo: [100px, 200px] to [150px, 250px]. Render the target image region in color and render other image regions in grayscale to obtain a landscape photo that highlights the target image region corresponding to the "sheep". Display the landscape photo on the front end to provide feedback to the user.

[0109] In this embodiment, the image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features, and the prompt text is input into the text encoding layer of the visual processing model to obtain text input features. The visual processing model also includes a feature processing layer and a decoding layer to ensure that the image input and text input are accurately represented in the model. The visual input features and text input features are input into the feature processing layer to perform the target visual task and obtain multimodal output features including multiple text feature units. The text feature units correspond to different regions in the image and represent the classification results of performing the target visual task on the image region. This completes the fine-grained parsing and classification of the image region, so that each text feature unit can accurately correspond to a specific region in the image and perform fine-grained target visual tasks on the image region. The multimodal output features are input into the decoding layer to locate the corresponding target image region in the image to be processed based on the multiple text feature units and obtain the target image that highlights the target image region. This not only ensures the accurate location and recognition of the target entity or attribute, but also greatly improves the visualization effect of the target image and intuitively shows the output structure of the model. This method does not add an extra task decoder to the model, accomplishes a variety of fine-grained perception visual tasks, does not increase the complexity of the model architecture, preserves the scalability of the model, and also promotes synergy between different visual tasks.

[0110] In one optional embodiment of this specification, the target visual task is a visual detection task, and any text feature unit represents the classification result of whether the corresponding image region in the image to be processed is a target detection entity; step 308 includes the following specific steps:

[0111] The multimodal output features are input into the decoding layer. Based on multiple text feature units, the coordinates of the target detection box on the target image region corresponding to the target detection entity are determined. Based on the coordinates of the target detection box, the target detection box is rendered on the target image region to obtain the target image that highlights the target image region.

[0112] Visual detection tasks are visual tasks that identify and locate specific categories of entities in an image. This is typically achieved by rendering bounding boxes to highlight the locations of these entities within the image. Common visual detection tasks include, but are not limited to: pedestrian detection, vehicle detection, lesion detection, face detection, road detection, and building detection.

[0113] Object detection entities are entities that need to be identified and located in visual detection tasks. Object detection entities can be any identifiable object, such as people, vehicles, animals, or other specific entities.

[0114] The target detection bounding box is a box used to highlight the target entity being detected. It can be a rectangular box, a polygonal box, a circular box, an elliptical box, an irregularly shaped box, etc.

[0115] The coordinates of the bounding box are used to determine the numerical coordinates of the bounding box's position in image space. For example, the coordinates of a rectangle are " <box>x_min, y_min, x_max, y_max< / box> ".

[0116] Based on the coordinate values ​​of the target detection box, the target detection box is rendered on the target image region to obtain a target image that highlights the target image region. One possible method is to convert the continuous coordinate values ​​of the target detection box into integer values ​​within a preset range, and then render the target detection box on the target image region based on the integer values ​​of the target detection box coordinates to obtain a target image that highlights the target image region.

[0117] For example, the text output feature h_t is input into the decoding layer. Based on multiple text feature units token, the continuous coordinate values ​​of the rectangular detection box covering the target image region corresponding to the target detection entity "sheep" are determined: (x_min: 56.345, y_min: 4.789, x_max: 96.567, y_max: 12.345). The continuous coordinate values ​​of the rectangular detection box are then converted into integer values ​​within a preset range. <box> 56，4，96，12< / box> Based on the integer values ​​of the coordinates of the rectangular detection box, a rectangular detection box is rendered on the target image region to obtain a landscape photo that highlights the target image region corresponding to the "sheep".

[0118] In the embodiments described in this specification, the detection boxes are accurately rendered on the image based on their coordinate values, achieving accurate localization and visualization of the target detection entities. This not only improves detection accuracy but also enhances the visual prominence of the target image region, enabling users to intuitively identify the target detection entities and complete visual detection. At the same time, no additional task decoders are added to the model, enabling various fine-grained perceptual visual detections without increasing the complexity of the model architecture, preserving the model's scalability, and promoting synergy between different visual tasks.

[0119] In one optional embodiment of this specification, the target visual task is a visual segmentation task. The multimodal output features include text output features and visual output features. The text output features include multiple text feature units, each of which includes a masked text feature unit representing a corresponding image region in the image to be processed as a target segmentation entity. The visual output features include visual feature units for each image region in the image to be processed. Step 308 includes the following specific steps:

[0120] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit is determined. The target image region represented by the target visual feature unit and corresponding to the target segmented entity is highlighted and rendered to obtain the target image.

[0121] Visual segmentation is a visual task that classifies and displays different regions of an image to achieve a fine-grained understanding of the image content. Visual segmentation not only requires identifying and locating objects in an image, but also accurately depicting the outlines and boundaries of these objects. Common visual segmentation tasks include, but are not limited to: semantic segmentation (e.g., in a city street scene photograph, sidewalks are segmented into the same color or label) and instance segmentation (e.g., autonomous vehicles need to accurately identify roads).

[0122] Target segmentation entities are entities that need to be identified and segmented in visual segmentation tasks. Target segmentation entities can be any identifiable object, such as people, vehicles, animals, or other specific entities.

[0123] The visual feature unit of an image region is the feature vector unit of the visual modality feature encoding vector of the visual segmentation task output by the visual processing model. Each visual feature unit corresponds to an image region and represents the visual information of the image region such as color, texture, and shape after performing the visual segmentation task.

[0124] The masked text feature unit of the target segmentation entity is the feature vector unit of the text modality feature encoding vector of the task result of the visual segmentation task, output by the visual processing model. Each masked text feature unit corresponds to a target segmentation entity, and the visual description information such as the shape, color, and size of the target segmentation entity is represented by a mask label. By adding masked text feature unit labels to the vocabulary of the visual processing model during the deep learning process, the model outputs masked text feature units when performing the visual segmentation task.

[0125] The feature similarity between masked text feature units and visual feature units is the vector similarity between the masked text feature units and visual feature units on the feature encoding vector. It is used to determine which image regions corresponding to visual feature units conform to the description of the target segmented entity. Feature similarity includes, but is not limited to: cosine similarity, Euclidean distance, dot product similarity, etc.

[0126] The target visual feature unit is a visual feature unit related to the visual characteristics of the target segmented entity.

[0127] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit is determined. The target image region represented by the target visual feature unit and corresponding to the target segmented entity is highlighted and rendered to obtain the target image. One possible approach is to input the text output features and visual output features into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit and the mask value of the target visual feature unit are determined. Based on the mask value, each image region in the image to be processed is distinguished and rendered to obtain the target image.

[0128] For example, the text output features h_t and visual output features h_v are input into the decoding layer, based on the masked text feature unit h_(t_token). <mask>The dot similarity between each visual feature unit h_(v_token) is used to determine the target visual feature unit h_(v_SEG) and the mask value of the target visual feature unit. The target image region corresponding to the target segmentation entity "sheep" in the landscape photo is located from [100px, 200px] to [150px, 250px]. Based on the mask value, the target image region is rendered in color style, and other image regions other than the target image region are rendered in grayscale. The resulting landscape photo highlights the target image region corresponding to "sheep". The landscape photo is then displayed on the front end and fed back to the user.

[0129] In the embodiments of this specification, feature retrieval is performed based on the feature similarity between the masked text feature units and each visual feature unit to determine the target visual feature unit. This achieves accurate localization and visualization of the target segmented entity, which not only improves the segmentation accuracy but also enhances the visual prominence of the target image region, enabling users to intuitively identify the target segmented entity. Visual segmentation is completed, avoiding the use of inefficient representations such as polygons. At the same time, no additional task decoder is added to the model, and various fine-grained perceptual visual detections are completed without increasing the complexity of the model architecture, preserving the scalability of the model, and promoting the synergistic effect between different visual tasks.

[0130] In one optional embodiment of this specification, text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit is determined. The target image region represented by the target visual feature unit and corresponding to the target segmented entity is highlighted and rendered to obtain the target image. The specific steps include the following:

[0131] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit and its binary mask are determined. Based on the binary mask of the target visual feature unit, each image region in the image to be processed is distinguished and rendered to obtain the target image.

[0132] The binary mask of the target visual feature unit is a mask marker value that distinguishes whether the image region corresponding to the target visual feature unit belongs to the target segmentation entity. It includes 0 and 1. When it is 0, it means that the image region corresponding to the target visual feature unit is the image region where the target segmentation entity is located. When it is 1, it means that the image region corresponding to the target visual feature unit is not the image region where the target segmentation entity is located.

[0133] For example, the text output features h_t and visual output features h_v are input into the decoding layer, based on the masked text feature unit h_(t_token). <mask>The dot similarity between the target visual feature unit and each visual feature unit h_(v_token) is used to determine the target visual feature unit and its binary mask as "0", thus locating the target image region corresponding to the target segmentation entity "sheep" in the landscape photo: [100px, 200px] to...

[0134] [150px, 250px], based on the binary mask "0", the target image area is rendered in color style, and other image areas other than the target image area are rendered in grayscale to obtain a landscape photo that highlights the target image area corresponding to "sheep", and the landscape photo is displayed on the front end to provide feedback to the user.

[0135] In the embodiments of this specification, binary masks are used to clearly distinguish and render each image region in the image to be processed, thereby obtaining a clear visualization of the target segmented entity and further enhancing the visual prominence of the target image region, enabling users to more intuitively identify the target segmented entity.

[0136] Because downsampling is often performed during the encoding of visual input features at the visual coding layer (e.g., the extracted image features are downsampled by 14 times compared to the original image), the mask obtained through the above feature retrieval is also downsampled and not refined enough. Therefore, the visual processing model needs to output a multi-valued mask of the target visual feature units to obtain a higher-resolution multi-valued mask image for distinguishable rendering, making the visual segmentation of the target entities in the target image more refined.

[0137] In one optional embodiment of this specification, text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit is determined. The target image region represented by the target visual feature unit and corresponding to the target segmented entity is highlighted and rendered to obtain the target image. The specific steps include the following:

[0138] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit and the multi-value mask of the target visual feature unit are determined. The multi-value masks of the target visual feature units are concatenated to obtain a multi-value mask image. The multi-value mask image is upsampled. Based on the upsampled multi-value mask image, each image region in the image to be processed is distinguished and rendered to obtain the target image.

[0139] The multi-valued mask for a target visual feature unit is a mask label value that distinguishes the probability that the image region corresponding to the target visual feature unit belongs to the target segmentation entity. It includes N values. N is a hyperparameter that needs to be output. <mask>The number of tokens is determined by predicting N^2 mask text feature units. After dot product calculation, the predicted mask size is N^2 x 16 x 16. After upsampling, the predicted mask size becomes (16N) x (16N), which provides higher and finer prediction resolution compared to binary masks. Different colors in the diagram represent the contribution of each mask text feature unit to the predicted mask image. In deep learning, the model learns to predict N^2 mask text feature units. For example, the default setting is N=4, predicting 16 mask text feature units. The choice of N was obtained through ablation experiments; N=4 achieves a balance between performance and speed.

[0140] A multivalued mask is an image-level mask constructed by stitching together multivalued masks based on the visual feature units of the target. The mask value of each image region represents the probability that the image region belongs to the target segmentation entity. Each value in the multivalued mask can be represented as an integer from 0 to N-1 (N is a hyperparameter), where different values ​​represent different levels of confidence or different classification results.

[0141] For example, the text output features h_t and visual output features h_v are input into the decoding layer, based on the masked text feature unit h_(t_token). <mask>The dot similarity between each visual feature unit h_(v_token) is used to determine the target visual feature unit and its sixteen-value mask, which includes "0", "1", "2", ..., "15". The target image region corresponding to the target segmentation entity "sheep" in the landscape photo is located from [100px, 200px] to [150px, 250px]. The sixteen-value mask "[10, 12, 3, 0, ...]; [7, 9, 4, 5, ...]; [11, 10, 13, 6, ...]; ..." is concatenated to obtain the sixteen-value mask image. The sixteen-value mask image is upsampled by 4 times. The upsampled sixteen-value mask image is aligned with the resolution of the original landscape photo through interpolation. The image regions in the original landscape photo are then distinguished and rendered to obtain a landscape photo with finely segmented target image regions corresponding to "sheep". This landscape photo is then displayed on the front end and fed back to the user.

[0142] In the embodiments of this specification, multi-value masks are used to further refine the segmentation and clearly distinguish each image region in the image to be processed, thereby obtaining a clear and detailed visualization of the target segmented entity, further enhancing the visual prominence of the target image region, and enabling users to more intuitively and clearly identify the target segmented entity.

[0143] Based on the above embodiments, Figure 4 shows a schematic diagram of the architecture of a visual task processing method provided by one embodiment of this specification, as shown in Figure 4:

[0144] Visual inspection task:

[0145] The stage of obtaining the image to be processed and the prompt text: Input an image to be processed containing the target entity, and the prompt text "Please detect the object in the image" to guide the model to perform the visual detection task.

[0146] Encoding stage: The image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features; the prompt text is input into the text encoding layer to obtain text input features.

[0147] Feature processing stage: Visual input features and text input features are input into the feature processing layer of the Transformer model for multimodal vision tasks, and text output features are output.

[0148] Decoding and Result Output Stage: The text output features are input into the decoding layer, and the coordinates of the predicted detection box covering the predicted image region corresponding to the detected entity are determined based on the text feature units. <box> 56，4，96，12< / box> The system then outputs a detection image with the target detection bounding box.

[0149] Binary mask visual segmentation task:

[0150] The stage of obtaining the image to be processed and the prompt text: The input is an image containing the target entity, and the prompt text "Please segment the object in the image" to guide the model to perform the visual segmentation task.

[0151] Encoding stage: The image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features; the prompt text is input into the text encoding layer to obtain text input features.

[0152] Feature processing stage: Visual input features and text input features are input into the feature processing layer of the Transformer model for multimodal vision tasks, and text output features and visual output features are output.

[0153] Decoding and Output: Input the text output features and visual output features into the decoding layer. Calculate the feature similarity between the masked text feature units in the input text output features and the visual output feature units in the visual output features through dot product. Generate a binary mask image based on the feature similarity. After interpolating and aligning the resolution of the mask image, perform differential rendering to obtain a segmented image that highlights the image region where the target entity is located.

[0154] Multi-value mask visual segmentation task:

[0155] The stage of obtaining the image to be processed and the prompt text: The input is an image containing the target entity, and the prompt text "Please segment the object in the image" to guide the model to perform the visual segmentation task.

[0156] Encoding stage: The image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features; the prompt text is input into the text encoding layer to obtain text input features.

[0157] Feature processing stage: Visual input features and text input features are input into the feature processing layer of the Transformer model for multimodal vision tasks, and text output features and visual output features are output.

[0158] Decoding and Output: Input the text output features and visual output features into the decoding layer. Calculate the feature similarity between the masked text feature units in the input text output features and the visual output feature units in the visual output features through dot product. Generate a mask image of a multi-valued mask based on the feature similarity. After interpolation and resolution alignment of the mask image, perform differential rendering to obtain a segmented image that highlights the image region where the target entity is located.

[0159] By improving the segmentation modeling approach and effectively utilizing image features from the model output through feature retrieval, the need for additional task decoders is avoided. Furthermore, detected bounding boxes are represented using text, unifying the main fine-grained perception tasks within the framework of the multimodal large language model itself, eliminating differences in modeling methods between different granularity perception tasks. Using the outputs of detection and segmentation tasks as intermediate outputs enhances the model's fine-grained image understanding, essentially achieving refined image understanding through a chain of thought processes.

[0160] Referring to Figure 5, which shows a flowchart of a training method for a visual processing model according to an embodiment of this specification, the method is applied to a model training platform and includes the following specific steps:

[0161] Step 502: Acquire a sample image, sample prompt text for performing the target visual task on the sample image, and a label image, wherein the label image area is highlighted in the label image.

[0162] The model training platform is an integrated system platform for training deep learning models, specifically designed for training neural network learning models, particularly in the field of vision tasks. It provides a complete set of tools and services, supporting the entire workflow from data preparation and model training to evaluation and deployment. Through this platform, users can efficiently build, train, and validate complex vision processing models to meet specific vision application needs.

[0163] The sample images are the original sample image data that are used to train the visual processing model and are required to perform visual task processing.

[0164] The sample prompt text is sample text data used to train the visual processing model and to guide the visual processing model to perform visual tasks.

[0165] Labeled images are image data that are labeled and used to train visual processing models to complete visual tasks.

[0166] The label image region is the image region in the label image that is labeled with a specific entity or attribute. This part of the image region represents the target region that the model needs to learn to recognize.

[0167] The acquisition of sample images, sample prompt text, and label images provides basic data support for subsequent model training.

[0168] Step 504: Input the sample image into the visual encoding layer of the initial visual processing model to obtain the sample visual input features, and input the prompt text into the text encoding layer of the initial visual processing model to obtain the sample text input features. The initial visual processing model also includes a feature processing layer and a decoding layer.

[0169] The initial visual processing model is an untrained visual processing model that requires targeted deep learning optimization for the target visual task. This initial model includes visual encoding layers, text encoding layers, feature processing layers, and decoding layers, aiming to perform various fine-grained visual perception tasks by combining image and text information. The initial model may be built on a certain infrastructure (such as Transformer) and may have already been pre-trained on a general dataset, but it still needs further tuning for the target visual task.

[0170] The visual input features of the samples are feature encoding vectors extracted from the sample images to describe the content of the images.

[0171] The sample text input features are feature encoding vectors extracted from the sample prompt text, which are used to describe the task content of the target visual task.

[0172] The sample images and prompt text are processed by visual encoding layers and text encoding layers respectively to generate feature vectors. This approach not only effectively extracts the core information from the input data but also provides multimodal support for subsequent tasks, enhancing the model's ability to handle complex visual tasks.

[0173] Step 506: Input the sample visual input features and sample text input features into the feature processing layer, perform the target visual task, and obtain the predicted multimodal output features including multiple text feature units, wherein any text feature unit represents the prediction classification result of the target visual task performed on the corresponding image region in the sample image.

[0174] The predicted multimodal output features are the multimodal feature encoding vectors of the task results of the target visual task output by the initial visual processing model, including but not limited to at least one of the following: predicted visual output features of visual modal and predicted text output features of text modal.

[0175] By combining visual and textual features, targeted visual tasks can be performed precisely on specific regions, improving prediction accuracy. Simultaneously, this multimodal fusion method helps capture the relationships between different modalities, thereby enhancing the overall model's perceptual capabilities.

[0176] Step 508: Input the predicted multimodal output features into the decoding layer, and locate the corresponding predicted image region in the sample image based on multiple text feature units.

[0177] The decoding layer uses text feature units to locate and predict image regions, achieving the transformation from features to specific image regions. This step is crucial for identifying and locating key regions in the image, providing accurate target region information for subsequent model training.

[0178] Step 510: Based on the predicted image region and the label image region, train the initial visual processing model to obtain the trained visual processing model.

[0179] The initial visual processing model is trained based on the predicted image region and the label image region. One possible approach is to calculate the loss value based on the predicted image region and the label image region, and then train the initial visual processing model based on the loss value to obtain the trained visual processing model. Further, the initial visual processing model is trained based on the loss value. Another possible approach is to adjust the model parameters of the initial visual processing model using a gradient update method based on the loss value. The model parameters can be only the parameters of the feature processing layer and the decoding layer, or they can be the parameters of the visual encoding layer, the text encoding layer, the feature processing layer, and the decoding layer; this is not limited here.

[0180] It should be noted that any details not described in detail in the embodiments of this specification can be found in the description of the technical solutions in the above embodiments, and will not be repeated here.

[0181] In the embodiments described in this specification, the performance of the initial visual processing model is improved in fine-grained visual perception tasks through precise training. At the same time, without adding additional task decoders to the model, a variety of fine-grained visual perception tasks are completed without increasing the complexity of the model architecture, preserving the scalability of the model, and promoting the synergistic effect between different visual tasks.

[0182] In one optional embodiment of this specification, the target visual task is a visual detection task, where any text feature unit characterizes whether the corresponding image region in the sample image is the predicted classification result of the sample detection entity, and a label detection box is overlaid on the label image region; step 508 includes the following specific steps:

[0183] The predicted multimodal output features are input into the decoding layer, and based on multiple text feature units, the coordinate values ​​of the predicted detection box on the predicted image region corresponding to the sample detection entity are determined.

[0184] Step 510 includes the following specific steps:

[0185] Based on the coordinates of the predicted detection boxes overlaid on the predicted image region and the coordinates of the label detection boxes overlaid on the label image region, the initial visual processing model is trained to obtain the trained visual processing model.

[0186] The label detection box is a box in the label image area used to highlight the detected entity in the sample.

[0187] The coordinates of the label detection box are used to determine the position of the label detection box in the image space.

[0188] The entities to be detected in a visual inspection task are those entities in the sample images and label images that need to be identified and located.

[0189] The predicted classification result of the sample detection entity is the output of the initial visual processing model, which classifies whether the corresponding image region in the sample image is the sample detection entity.

[0190] The predicted detection bounding box is a box output by the initial visual processing model that is used to highlight the detected entity in the predicted image region.

[0191] The coordinates of the predicted detection box are used to determine the position of the predicted detection box in the image space.

[0192] The initial visual processing model is trained based on the coordinates of the predicted detection boxes overlaid on the prediction image region and the coordinates of the label detection boxes overlaid on the label image region. One possible approach is to calculate a loss value based on these coordinates, and then train the initial visual processing model using this loss value to obtain a trained visual processing model. Further, the initial visual processing model is trained based on the loss value. Another possible approach is to adjust the model parameters of the initial visual processing model using a gradient update method based on the loss value. These model parameters can be solely the parameters of the feature processing layer and the decoding layer, or they can include parameters of the visual encoding layer, the text encoding layer, the feature processing layer, and the decoding layer; no limitation is imposed here.

[0193] It should be noted that any details not described in detail in the embodiments of this specification can be found in the description of the technical solutions in the above embodiments.

[0194] In the embodiments described in this specification, the performance of the model in fine-grained visual perception visual detection tasks is improved by accurately training the initial visual processing model. At the same time, without adding additional task decoders to the model, it completes a variety of fine-grained visual perception visual detection tasks, without increasing the complexity of the model architecture, retaining the scalability of the model, and promoting the synergistic effect between different visual tasks.

[0195] In one optional embodiment of this specification, the target visual task is a visual segmentation task. The predicted multimodal output features include predicted text output features and predicted visual output features. The predicted text output features include multiple text feature units, each of which includes a mask text feature unit representing a segmented entity in the corresponding image region of the sample image. The predicted visual output features include visual feature units for each image region in the sample image, and the label image region corresponds to a label visual feature unit. Step 508 includes the following specific steps:

[0196] The predicted multimodal output features are input into the decoding layer, and the predicted visual feature units of the predicted image region are determined based on the feature similarity between the masked text feature units and each visual feature unit.

[0197] Step 510 includes the following specific steps:

[0198] The initial visual processing model is trained based on the predicted visual feature units of the predicted image region and the labeled visual feature units of the labeled image region to obtain the trained visual processing model.

[0199] The sample segmentation entity is the entity that needs to be identified and segmented from the sample image and label image in the visual detection task.

[0200] The visual feature unit of an image region is the feature vector unit of the visual modality feature encoding vector of the task result of the visual segmentation task, which is output by the initial visual processing model.

[0201] The masked text feature unit of the sample segmented entity is the feature vector unit of the text modality feature encoding vector of the task result of the visual segmentation task output by the initial visual processing model.

[0202] The feature similarity between masked text feature units and visual feature units is the vector similarity between the masked text feature units and visual feature units on the feature encoding vector.

[0203] The predicted visual feature unit is the visual feature unit related to the visual characteristics of the entity being segmented in the sample.

[0204] The predicted multimodal output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the predicted visual feature units of the predicted image region are determined. Alternatively, the predicted multimodal output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the predicted visual feature units of the predicted image region and the mask values ​​of the predicted visual feature units are determined.

[0205] Furthermore, based on the feature similarity between the masked text feature units and each visual feature unit, the predicted visual feature units and the mask values ​​of the predicted visual feature units of the predicted image region are determined. One possible approach is to determine the predicted visual feature units and the binary masks of the predicted visual feature units of the predicted image region based on the feature similarity between the masked text feature units and each visual feature unit.

[0206] Furthermore, based on the feature similarity between the masked text feature units and each visual feature unit, the predicted visual feature units and the mask values ​​of the predicted visual feature units of the predicted image region are determined. One possible approach is to determine the predicted visual feature units and the multi-value masks of the predicted visual feature units of the predicted image region based on the feature similarity between the masked text feature units and each visual feature unit.

[0207] It should be noted that any details not described in detail in the embodiments of this specification can be found in the description of the technical solutions in the above embodiments.

[0208] In the embodiments described in this specification, the performance of the model in fine-grained visual perception visual segmentation tasks is improved by accurately training the initial visual processing model. At the same time, without adding an additional task decoder to the model, it completes a variety of fine-grained visual perception visual segmentation tasks, without increasing the complexity of the model architecture, retaining the scalability of the model, and promoting the synergistic effect between different visual tasks.

[0209] Referring to Figure 6, which shows a flowchart of an image generation method provided in one embodiment of this specification, the method is applied to a vision task platform and includes the following specific steps:

[0210] Step 602: Obtain the first prompt text for generating the target image that highlights the target image region.

[0211] The Visual Task Platform is an integrated visual service system platform that utilizes deep learning and computer vision technologies to handle various visual tasks, such as image generation, object detection, segmentation, and classification. The platform deploys visual processing models and provides external interfaces, enabling users to guide the models to perform complex visual processing tasks through text prompts or other forms of input.

[0212] The first prompt text is text data used to guide the visual processing model in performing the image generation task. It contains basic information or objectives about the image to be generated. This text guides the visual processing model to understand the user's needs and generate the corresponding image to be processed and the second prompt text based on these needs. For example, the first prompt text might be "Please generate a landscape photo containing sheep." This text provides the visual processing model with initial direction and contextual information, helping the model generate the required image to be processed and the second prompt text.

[0213] For example, a user wants to create an image of a specific scene through a visual task platform and perform a detailed visual analysis on it. The user can first enter the initial prompt text "Please generate a landscape photo containing sheep".

[0214] The first prompt text of the target image used to generate the target image region that is highlighted is obtained, which provides data support for the subsequent generation of the image to be processed and the second prompt text.

[0215] Step 604: Using a visual processing model, perform an image generation task based on the first prompt text to generate an image to be processed and a second prompt text for performing a target visual task on the image to be processed. Then, perform the target visual task on the image to be processed based on the second prompt text to obtain the target image. The step of performing the target visual task implements the steps of the above-mentioned visual task processing method.

[0216] The image generation task is a multimodal generation task that utilizes a visual processing model to generate an image based on the first prompt text.

[0217] The second cue text is text data used to guide the visual processing model in performing visual tasks. It guides the model on how to perform visual tasks and provides additional contextual information for processing images.

[0218] For example, after receiving this instruction, the vision task platform uses its built-in vision processing model to generate an image that meets the requirements. Then, to further refine the processing, the platform automatically generates a second prompt text, such as "Please identify and label all the sheep in this picture." Based on this second prompt text, the model performs precise target vision task processing on the generated image, and finally outputs a target image that not only contains the required content but has also been analyzed and labeled in detail.

[0219] It should be noted that any details not described in detail in the embodiments of this specification can be found in the description of the technical solutions in the above embodiments, and will not be repeated here.

[0220] In the embodiments described in this specification, based on the first prompt text indicating user requirements, the visual processing model can efficiently generate an image to be processed that meets the requirements; based on the second prompt text generated according to the specific task requirements of the image, the model further performs a precise target visual task, thereby obtaining the target image. This not only improves the accuracy and efficiency of image generation, but also greatly enhances the flexibility and targeting of image processing, meeting diverse visual task needs.

[0221] The following description, in conjunction with Figure 7, uses the application of the visual task processing method provided in this specification in a multimodal intelligent question-answering scenario as an example to further illustrate the visual task processing method. Figure 7 shows a flowchart of the processing procedure of a visual task processing method applied to a multimodal intelligent question-answering scenario according to an embodiment of this specification, including the following specific steps:

[0222] Step 702: Obtain the road detection image and the prompt text "Identify how many people are in the image" for performing a visual detection task on the road detection image.

[0223] Step 704: Input the road detection image into the visual segmenter of the visual processing model to encode and obtain visual input features, and input the prompt text into the text segmenter of the visual processing model to encode and obtain text input features. The visual processing model also includes a Transformer model for multimodal visual tasks and a decoding layer.

[0224] Step 706: Input the visual input features and text input features into the Transformer model of the multimodal vision task, perform the visual detection task, and obtain multimodal output features including multiple text feature units, wherein any text feature unit represents the classification result of the visual detection task performed on the corresponding image region in the road detection image.

[0225] Step 708: Input the text output features and visual output features into the decoding layer. Based on multiple text feature units, determine the coordinate values ​​of the target detection boxes covering the target image region corresponding to the target detection entity. Based on the coordinate values ​​of the target detection boxes, render the target detection boxes on the target image region to obtain a segmented image that highlights the target image region. Determine the number of pedestrians based on the number of target detection boxes.

[0226] In the embodiments of this specification, the detection boxes are accurately rendered on the image based on the coordinate values ​​of the target detection boxes, realizing accurate localization and visualization of the target detection entities. This not only improves the detection accuracy but also enhances the visual prominence of the target image region, enabling users to intuitively identify pedestrians and complete pedestrian quantity detection. It also reduces the illusion problem in image understanding. At the same time, it does not add an additional task decoder to the model, completes various fine-grained visual detections, does not increase the complexity of the model architecture, retains the scalability of the model, and promotes the synergy between different visual tasks.

[0227] Corresponding to the above method embodiments, this specification also provides a vision task platform embodiment. Figure 8 shows a schematic diagram of the structure of a vision task platform provided in one embodiment of this specification. As shown in Figure 8, the vision task platform 800 includes a vision processing interface 810 and a response unit 820;

[0228] The visual processing interface 810 is used to acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed;

[0229] The response unit 820 is used to execute the steps of the above-described visual task processing method.

[0230] The Visual Task Platform 800 is an integrated visual service system platform that utilizes deep learning and computer vision technologies to handle various visual tasks, such as image generation, object detection, segmentation, and classification. The Visual Task Platform 800 deploys visual processing models.

[0231] The visual processing interface 810 is a front-end component of the visual task platform 800, used to interact with external systems, receive user-submitted images to be processed, and prompt text for performing specific visual tasks on those images. The visual processing interface 810 supports various image input formats and can parse task requirements described in natural language, such as object recognition and scene classification. Through efficient API calls, the visual processing interface ensures rapid data transmission and processing, improving the user experience.

[0232] The response unit 820 is the core processing module of the vision task platform 800, implementing a series of complex algorithms to perform vision tasks. The response unit 820 also includes a feedback mechanism to evaluate the performance of the task and optimize subsequent processing strategies accordingly. This enables the vision task platform 800 to provide accurate and reliable services in various application scenarios.

[0233] In one optional embodiment of this specification, the visual processing interface 810 is further configured to acquire first prompt text for generating a target image that highlights a target image region;

[0234] The response unit 820 is also configured to utilize a visual processing model to perform an image generation task based on the first prompt text, generating an image to be processed and a second prompt text for performing a target visual task on the image to be processed.

[0235] In the embodiments described in this specification, the visual processing model deployed on the visual task platform does not add an additional task decoder, completes a variety of fine-grained perception visual tasks, does not increase the complexity of the model architecture, retains the scalability of the model, and also promotes the synergy between different visual tasks.

[0236] The above is an illustrative scheme of a visual task platform according to this embodiment. It should be noted that the technical solution of this visual task platform and the technical solution of the above-described visual task processing method belong to the same concept. For details not described in detail in the technical solution of the visual task platform, please refer to the description of the technical solution of the above-described visual task processing method.

[0237] Corresponding to the above method embodiments, this specification also provides a model training platform embodiment. Figure 9 shows a schematic diagram of the structure of a model training platform provided in one embodiment of this specification. As shown in Figure 9, the model training platform 900 includes a model training interface 910 and a response unit 920;

[0238] The model training interface 910 is used to acquire sample images, sample prompt text for performing target visual tasks on the sample images, and label images, wherein the label image area is highlighted in the label image;

[0239] The response unit 920 is used to perform the steps of the training method of the above-described visual processing model.

[0240] Model Training Platform 900 is an integrated system platform for training deep learning models, specifically designed for training neural network learning models, particularly in the field of vision tasks. Model Training Platform 900 provides a complete suite of tools and services, supporting the entire workflow from data preparation and model training to evaluation and deployment.

[0241] The Model Training Interface 910 is a front-end component of the Model Training Platform 900, used for interaction with external systems. The Model Training Interface 910 is used to acquire sample images, sample prompt text, and label images. This data is crucial for training high-quality visual processing models. The Model Training Interface 910 allows users to upload datasets in various ways, ensuring data security and integrity. Furthermore, the Model Training Interface 910 supports automated data augmentation to increase the diversity of training data, thereby improving the model's generalization ability.

[0242] The response unit 920 is the core processing module of the model training platform 900, implementing a series of complex algorithms to perform visual tasks. The response unit 920 also includes a feedback mechanism to evaluate the effectiveness of model training and optimize subsequent processing strategies accordingly. This enables the visual task platform 800 to provide accurate and reliable services in various application scenarios.

[0243] In the embodiments described in this specification, the performance of the initial visual processing model is improved in fine-grained visual perception tasks through precise training. At the same time, without adding additional task decoders to the model, a variety of fine-grained visual perception tasks are completed without increasing the complexity of the model architecture, preserving the scalability of the model, and promoting the synergistic effect between different visual tasks.

[0244] The above is an illustrative scheme of a model training platform according to this embodiment. It should be noted that the technical solution of this model training platform and the technical solution of the above-described visual processing model training method belong to the same concept. For details not described in detail in the technical solution of the model training platform, please refer to the description of the technical solution of the above-described visual processing model training method.

[0245] Corresponding to the above method embodiments, this specification also provides embodiments of a visual task processing device. Figure 10 shows a schematic diagram of the structure of a visual task processing device provided in one embodiment of this specification. As shown in Figure 10, the device includes:

[0246] The first acquisition module 1002 is configured to acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed;

[0247] The first encoding module 1004 is configured to input the image to be processed into the visual encoding layer of the visual processing model, encode to obtain visual input features, and input the prompt text into the text encoding layer of the visual processing model, encode to obtain text input features. The visual processing model also includes a feature processing layer and a decoding layer.

[0248] The first processing module 1006 is configured to input visual input features and text input features into the feature processing layer, perform the target visual task, and obtain multimodal output features including multiple text feature units, wherein any text feature unit represents the classification result of the target visual task performed on the corresponding image region in the image to be processed;

[0249] The first decoding module 1008 is configured to input multimodal output features into the decoding layer, locate the corresponding target image region in the image to be processed based on multiple text feature units, and obtain a target image that highlights the target image region.

[0250] Optionally, the target visual task is a visual detection task, where any text feature unit represents the classification result of whether the corresponding image region in the image to be processed is a target detection entity; the first decoding module 1008 is further configured as follows:

[0251] The multimodal output features are input into the decoding layer. Based on multiple text feature units, the coordinates of the target detection box on the target image region corresponding to the target detection entity are determined. Based on the coordinates of the target detection box, the target detection box is rendered on the target image region to obtain the target image that highlights the target image region.

[0252] Optionally, the target visual task is a visual segmentation task, and the multimodal output features include text output features and visual output features. The text output features include multiple text feature units, each of which includes a masked text feature unit representing a corresponding image region in the image to be processed as a target segmentation entity. The visual output features include visual feature units for each image region in the image to be processed. The first decoding module 1008 is further configured as follows:

[0253] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit is determined. The target image region represented by the target visual feature unit and corresponding to the target segmented entity is highlighted and rendered to obtain the target image.

[0254] Optionally, the first decoding module 1008 is further configured as follows:

[0255] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the binary mask of each visual feature unit is determined. Based on the binary mask of each visual feature unit, each image region in the image to be processed is distinguished and rendered to obtain the target image.

[0256] Optionally, the first decoding module 1008 is further configured as follows:

[0257] The text output features and visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the multi-value mask of each visual feature unit is determined. The multi-value masks of each visual feature unit are concatenated to obtain a multi-value mask image. The multi-value mask image is upsampled. Based on the upsampled multi-value mask image, each image region in the image to be processed is distinguished and rendered to obtain the target image.

[0258] In the embodiments described in this specification, no additional task decoders were added to the model, which accomplished a variety of fine-grained perception visual tasks without increasing the complexity of the model architecture, preserving the scalability of the model, and promoting synergy between different visual tasks.

[0259] The above is an illustrative scheme of a visual task processing device according to this embodiment. It should be noted that the technical solution of this visual task processing device and the technical solution of the above-described visual task processing method belong to the same concept. For details not described in detail in the technical solution of the visual task processing device, please refer to the description of the technical solution of the above-described visual task processing method.

[0260] Corresponding to the above method embodiments, this specification also provides an embodiment of a training device for a visual processing model. Figure 11 shows a schematic diagram of the structure of a training device for a visual processing model provided in one embodiment of this specification. As shown in Figure 11, the device is applied to a model training platform and includes:

[0261] The second acquisition module 1102 is configured to acquire a sample image, sample prompt text for performing a target visual task on the sample image, and a label image, wherein the label image has a highlighted label image area.

[0262] The second encoding module 1104 is configured to input the sample image into the visual encoding layer of the initial visual processing model to encode and obtain the sample visual input features, and input the prompt text into the text encoding layer of the initial visual processing model to encode and obtain the sample text input features. The initial visual processing model also includes a feature processing layer and a decoding layer.

[0263] The second processing module 1106 is configured to input the sample visual input features and sample text input features into the feature processing layer, perform the target visual task, and obtain the predicted multimodal output features including multiple text feature units, wherein any text feature unit represents the predicted classification result of performing the target visual task on the corresponding image region in the sample image.

[0264] The second decoding module 1108 is configured to input the predicted multimodal output features into the decoding layer and locate the corresponding predicted image region in the sample image based on multiple text feature units;

[0265] The second training module 1110 is configured to train the initial visual processing model based on the predicted image region and the label image region to obtain the trained visual processing model.

[0266] Optionally, the target visual task is a visual detection task, where any text feature unit represents the predicted classification result of whether the corresponding image region in the sample image is the sample detection entity, and a label detection box is overlaid on the label image region; the second decoding module 1108 is further configured as follows:

[0267] The predicted multimodal output features are input into the decoding layer, and based on multiple text feature units, the coordinate values ​​of the predicted detection box on the predicted image region corresponding to the sample detection entity are determined.

[0268] Correspondingly, the second training module 1110 is further configured as follows:

[0269] Based on the coordinates of the predicted detection boxes overlaid on the predicted image region and the coordinates of the label detection boxes overlaid on the label image region, the initial visual processing model is trained to obtain the trained visual processing model.

[0270] Optionally, the target visual task is a visual segmentation task. The predicted multimodal output features include predicted text output features and predicted visual output features. The predicted text output features include multiple text feature units, each of which includes a masked text feature unit representing a segmented entity in the corresponding image region of the sample image. The predicted visual output features include visual feature units for each image region in the sample image, and the label image region corresponds to a label visual feature unit. The second decoding module 1108 is further configured as follows:

[0271] The predicted multimodal output features are input into the decoding layer, and the predicted visual feature units of the predicted image region are determined based on the feature similarity between the masked text feature units and each visual feature unit.

[0272] Correspondingly, the second training module 1110 is further configured as follows:

[0273] The initial visual processing model is trained based on the predicted visual feature units of the predicted image region and the labeled visual feature units of the labeled image region to obtain the trained visual processing model.

[0274] In the embodiments described in this specification, the performance of the initial visual processing model is improved in fine-grained visual perception tasks through precise training. At the same time, without adding additional task decoders to the model, a variety of fine-grained visual perception tasks are completed without increasing the complexity of the model architecture, preserving the scalability of the model, and promoting the synergistic effect between different visual tasks.

[0275] The above is a schematic scheme of a training device for a visual processing model according to this embodiment. It should be noted that the technical solution of this training device for a visual processing model and the technical solution of the above-described training method for a visual processing model belong to the same concept. For details not described in detail in the technical solution of the training device for a visual processing model, please refer to the description of the technical solution of the above-described training method for a visual processing model.

[0276] Corresponding to the above method embodiments, this specification also provides an image generation apparatus embodiment. Figure 12 shows a schematic diagram of an image generation apparatus provided in one embodiment of this specification. As shown in Figure 12, the apparatus is applied to a vision task platform and includes:

[0277] The third acquisition module 1202 is configured to acquire first prompt text for generating a target image that highlights the target image region;

[0278] The generation module 1204 is configured to use a visual processing model to perform an image generation task based on a first prompt text to generate an image to be processed and a second prompt text for performing a target visual task on the image to be processed, and to perform a target visual task on the image to be processed based on the second prompt text to obtain a target image, wherein the step of performing the target visual task implements the steps of the above-mentioned visual task processing method.

[0279] In the embodiments described in this specification, based on the first prompt text indicating user requirements, the visual processing model can efficiently generate an image to be processed that meets the requirements; based on the second prompt text generated according to the specific task requirements of the image, the model further performs a precise target visual task, thereby obtaining the target image. This not only improves the accuracy and efficiency of image generation, but also greatly enhances the flexibility and targeting of image processing, meeting diverse visual task needs.

[0280] The above is an illustrative scheme of an image generation apparatus according to this embodiment. It should be noted that the technical solution of this image generation apparatus and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the image generation apparatus, please refer to the description of the technical solution of the image generation method described above.

[0281] Figure 13 shows a structural block diagram of a computing device according to an embodiment of this specification. The components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and a database 1350 is used to store data.

[0282] The computing device 1300 also includes an access device 1340, which enables the computing device 1300 to communicate via one or more networks 1360. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1340 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0283] In one embodiment of this specification, the aforementioned components of the computing device 1300, as well as other components not shown in FIG. 13, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 13 is merely for illustrative purposes and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0284] The computing device 1300 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1300 can also be a mobile or stationary server.

[0285] The processor 1320 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned visual task processing method, visual processing model training method, or image generation method.

[0286] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the above-described visual task processing method, visual processing model training method, and image generation method. For details not described in detail in the technical solution of the computing device, please refer to the descriptions of the technical solutions of the above-described visual task processing method, visual processing model training method, or image generation method.

[0287] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual task processing method, visual processing model training method, or image generation method.

[0288] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the above-described visual task processing method, visual processing model training method, and image generation method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the above-described visual task processing method, visual processing model training method, or image generation method.

[0289] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual task processing method, visual processing model training method, or image generation method.

[0290] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned visual task processing method, visual processing model training method, and image generation method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-mentioned visual task processing method, visual processing model training method, or image generation method.

[0291] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0292] The computer instructions include computer program code, which may be in the form of source code, physical code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0293] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0294] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0295] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. A visual task processing method, comprising: Acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed; The image to be processed is input into the visual encoding layer of the visual processing model to obtain visual input features, and the prompt text is input into the text encoding layer of the visual processing model to obtain text input features. The visual processing model further includes a feature processing layer and a decoding layer. The visual input features and the text input features are input into the feature processing layer to perform the target visual task and obtain multimodal output features including multiple text feature units, wherein any text feature unit represents the classification result of performing the target visual task on the corresponding image region in the image to be processed; The multimodal output features are input into the decoding layer, and the corresponding target image region in the image to be processed is located based on the multiple text feature units to obtain a target image that highlights the target image region.

2. The method according to claim 1, wherein the target visual task is a visual detection task, and any text feature unit represents the classification result of whether the corresponding image region in the image to be processed is a target detection entity; The step of inputting the multimodal output features into the decoding layer, locating the corresponding target image region in the image to be processed based on the multiple text feature units, and obtaining a target image that highlights the target image region includes: The multimodal output features are input into the decoding layer. Based on the multiple text feature units, the coordinates of the target detection box covering the target image region corresponding to the target detection entity are determined. Based on the coordinates of the target detection box, the target detection box is rendered on the target image region to obtain a target image that highlights the target image region.

3. The method according to claim 1, wherein the target visual task is a visual segmentation task, the multimodal output features include text output features and visual output features, the text output features include the plurality of text feature units, the plurality of text feature units include mask text feature units characterizing the corresponding image region in the image to be processed as the target segmentation entity, and the visual output features include visual feature units of each image region in the image to be processed; The step of inputting the multimodal output features into the decoding layer, locating the corresponding target image region in the image to be processed based on the multiple text feature units, and obtaining a target image that highlights the target image region includes: The text output features and the visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the target visual feature unit is determined. The target image region represented by the target visual feature unit and corresponding to the target segmented entity is highlighted and rendered to obtain the target image.

4. The method according to claim 3, wherein inputting the text output features and the visual output features into the decoding layer, and determining the target visual feature unit based on the feature similarity between the masked text feature unit and each visual feature unit, comprises: The text output features and the visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, the binary mask of each visual feature unit is determined. Based on the binary mask of each visual feature unit, each image region in the image to be processed is distinguished and rendered to obtain the target image.

5. The method according to claim 3, wherein inputting the text output features and the visual output features into the decoding layer, and determining the target visual feature unit based on the feature similarity between the masked text feature unit and each visual feature unit, comprises: The text output features and the visual output features are input into the decoding layer. Based on the feature similarity between the masked text feature units and each visual feature unit, a multi-value mask for each visual feature unit is determined. The multi-value masks of each visual feature unit are concatenated to obtain a multi-value mask image. The multi-value mask image is upsampled. Based on the upsampled multi-value mask image, each image region in the image to be processed is distinguished and rendered to obtain the target image.

6. The method according to any one of claims 1 to 5, wherein acquiring the image to be processed and the prompt text for performing a target visual task on the image to be processed comprises: Receive the image to be processed sent by the front end, and the prompt text for performing the target visual task on the image to be processed; or Obtain the image to be processed from a predefined dataset, and prompt text for performing a target visual task on the image to be processed; or The image to be processed and the prompt text for performing the target visual task on the image to be processed are obtained through the data interface.

7. The method according to any one of claims 1 to 6, further comprising, before inputting the visual input features and the text input features into the feature processing layer, performing the target visual task, and obtaining multimodal output features including multiple text feature units: By concatenating the visual input features and the text input features, multimodal input features are obtained; Correspondingly, the step of inputting the visual input features and the text input features into the feature processing layer, performing the target visual task, and obtaining multimodal output features including multiple text feature units includes: The multimodal input features are input into the feature processing layer to perform the target vision task and obtain multimodal output features including multiple text feature units.

8. The method according to any one of claims 3 to 5, wherein the feature similarity includes at least one of cosine similarity, Euclidean distance, and dot product similarity.

9. A training method for a visual processing model, applied to a model training platform, comprising: Acquire a sample image, sample prompt text for performing a target visual task on the sample image, and a label image, wherein the label image has a highlighted label image area; The sample image is input into the visual encoding layer of the initial visual processing model to obtain sample visual input features, and the prompt text is input into the text encoding layer of the initial visual processing model to obtain sample text input features. The initial visual processing model further includes a feature processing layer and a decoding layer. The sample visual input features and the sample text input features are input into the feature processing layer to perform the target visual task and obtain a predicted multimodal output feature including multiple text feature units, wherein any text feature unit represents the predicted classification result of performing the target visual task on the corresponding image region in the sample image; The predicted multimodal output features are input into the decoding layer, and the corresponding predicted image region in the sample image is located based on the multiple text feature units; Based on the predicted image region and the labeled image region, the initial visual processing model is trained to obtain a trained visual processing model.

10. The method according to claim 9, wherein the target visual task is a visual detection task, any text feature unit characterizes whether the corresponding image region in the sample image is the predicted classification result of the sample detection entity, and the label image region is covered with a label detection box; The step of inputting the predicted multimodal output features into the decoding layer and locating the corresponding predicted image region in the sample image based on the multiple text feature units includes: The predicted multimodal output features are input into the decoding layer, and based on the multiple text feature units, the coordinate values ​​of the predicted detection box on the predicted image region corresponding to the sample detection entity are determined. The step of training the initial visual processing model based on the predicted image region and the labeled image region includes: Based on the coordinates of the predicted detection box covering the predicted image region and the coordinates of the label detection box covering the label image region, the initial visual processing model is trained to obtain a trained visual processing model.

11. The method according to claim 9, wherein the target visual task is a visual segmentation task, the predicted multimodal output features include predicted text output features and predicted visual output features, the predicted text output features include the plurality of text feature units, the plurality of text feature units include mask text feature units characterizing the corresponding image region in the sample image as a sample segmentation entity, the predicted visual output features include visual feature units of each image region in the sample image, and the label image region corresponds to a label visual feature unit; The step of inputting the predicted multimodal output features into the decoding layer and locating the corresponding predicted image region in the sample image based on the multiple text feature units includes: The predicted multimodal output features are input into the decoding layer, and the predicted visual feature units of the predicted image region are determined based on the feature similarity between the masked text feature units and each visual feature unit. The step of training the initial visual processing model based on the predicted image region and the labeled image region includes: Based on the predicted visual feature units of the predicted image region and the label visual feature units of the label image region, the initial visual processing model is trained to obtain the trained visual processing model.

12. The method according to any one of claims 9 to 11, wherein training the initial visual processing model based on the predicted image region and the label image region to obtain a trained visual processing model comprises: The loss value is calculated based on the predicted image region and the labeled image region; Based on the loss value, the model parameters of the initial visual processing model are adjusted using the gradient update method. The model parameters include the parameters of the feature processing layer and the decoding layer, or the parameters of the visual encoding layer, the text encoding layer, the feature processing layer, and the decoding layer.

13. An image generation method, applied to a vision task platform, comprising: Obtain the first prompt text used to generate the target image that highlights the target image region; Using a visual processing model, an image generation task is performed based on the first prompt text to generate an image to be processed and a second prompt text for performing a target visual task on the image to be processed. The target visual task is then performed on the image to be processed based on the second prompt text to obtain the target image. The step of performing the target visual task implements the steps of the visual task processing method according to any one of claims 1 to 5.

14. The method according to claim 13, wherein the step of using a visual processing model to perform an image generation task based on the first prompt text to generate an image to be processed and a second prompt text for performing a target visual task on the image to be processed includes: The image to be processed is generated based on the first prompt text using the image generation module in the visual processing model. Using the prompt text generation module in the visual processing model, the second prompt text is generated based on the first prompt text and the image to be processed.

15. A vision task platform, comprising a vision processing interface and a response unit; The visual processing interface is used to acquire the image to be processed and the prompt text for performing the target visual task on the image to be processed; The response unit is used to perform the steps of the visual task processing method according to any one of claims 1 to 8.

16. The visual task platform according to claim 15, wherein the visual processing interface is further configured to acquire first prompt text for generating a target image that highlights the target image region; The response unit is further configured to utilize a visual processing model to perform an image generation task based on the first prompt text, generating an image to be processed and a second prompt text for performing a target visual task on the image to be processed.

17. A model training platform, comprising a model training interface and a response unit; The model training interface is used to acquire sample images, sample prompt text for performing a target visual task on the sample images, and label images, wherein... The label image area is highlighted in the label image; The response unit is configured to perform the steps of the training method for the visual processing model according to any one of claims 9-12.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

Images