Image processing method and apparatus, device, medium, and program product

Abstract

Disclosed in embodiments of the present application are an image processing method and apparatus, a device, a medium, and a program product. The method comprises: in response to a content understanding task in a service scenario, acquiring a task rule of the content understanding task; acquiring an image feature vector of an image; and performing content understanding processing on the image feature vector according to the task rule to obtain a content understanding result of the image. The use of the embodiments of the present application significantly reduces the processing costs and technical threshold for image content understanding, and improves the efficiency of image content understanding.

Description

An image processing method, apparatus, device, medium, and program product

[0001] This application claims priority to Chinese Patent Application No. 2025101243206, filed on January 26, 2025, entitled “An Image Processing Method, Apparatus, Device, Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence, specifically to an image processing method, an image processing apparatus, a computer-readable storage medium, and a computer program product. Background Technology

[0003] Image content understanding (IU) refers to the process of recognizing and understanding the content of an image.

[0004] Currently, mainstream image content understanding technologies offer relatively complex, costly, and high-barrier-to-entry methods for understanding image content. For example, developing dedicated end-to-end models for image content understanding is not only time-consuming (e.g., months or weeks), but also requires specialized algorithm engineers, resulting in a high development threshold and low efficiency in image content understanding. Furthermore, the developed end-to-end models can only identify a single content category, leading to the need to develop numerous end-to-end models and significantly increasing the cost of image content understanding. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, medium, and program product that significantly reduces the processing cost and technical threshold of image content understanding and improves the efficiency of image content understanding.

[0006] On one hand, embodiments of this application provide an image processing method, which includes:

[0007] In response to content understanding tasks in business scenarios, the task rules for the content understanding tasks are obtained. The content understanding tasks are used to indicate the content understanding processing of images. The task rules are used to indicate the processing rules used when performing content understanding processing. The task rules are pre-configured based on prompt words or fine-tuning models.

[0008] Obtain the image feature vector of the image; the image feature vector is used to characterize the image features.

[0009] The image feature vectors are processed for content understanding according to the task rules to obtain the content understanding result of the image. The content understanding result is used to indicate the content category to which the image belongs.

[0010] On the other hand, embodiments of this application provide an image processing apparatus, which includes: a content understanding module, a rule module, and a basis model inference module;

[0011] The content understanding module is used to respond to content understanding tasks in business scenarios. It obtains the task rules for the content understanding task from the rule module. The content understanding task is used to instruct the image to be processed for content understanding. The task rules are used to instruct the processing rules to be used when performing content understanding processing. The task rules are pre-configured based on prompt words or fine-tuning models.

[0012] The basis model inference module is used to obtain the image feature vector of the image, which is used to characterize the image features of the image;

[0013] The content understanding module is also used to perform content understanding processing on the image feature vectors according to the task rules, and obtain the content understanding result of the image. The content understanding result is used to indicate the content category to which the image belongs.

[0014] On the other hand, embodiments of this application provide a computer device, which includes:

[0015] A processor is used to load and execute computer programs;

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image processing method.

[0017] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described image processing method.

[0018] On the other hand, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the above-described image processing method.

[0019] In this embodiment, business users can develop prompts and fine-tune models through an image content understanding platform, and then pre-configure the fine-tuned models obtained from the prompts and model fine-tuning into the task rules of specific content understanding tasks. This method, which allows business users to directly develop prompts and fine-tune models within the image content understanding system, eliminates the need for specialized algorithmic capabilities, effectively lowering the barrier to entry for image content understanding and significantly reducing development costs. During the image content understanding process, when a content understanding task instructs the processing of an image, the image feature vector of the image to be processed is obtained. This image feature vector represents the image features and can better express the semantics of the image content. Furthermore, the task rules for this content understanding task are also directly obtained. Since the task rules, pre-configured based on prompts or fine-tuned models, indicate the processing rules used when processing the image content, the image feature vector can be directly processed according to the task rules, simplifying the complexity of image content understanding processing. Specifically, based on the processing rules indicated by the task rules and the image feature vector, it determines whether the image contains the content category indicated by the prompt words in the processing rules or the content category that the fine-tuning model can recognize, thus obtaining the image content understanding result. For the same image, only the image feature vector needs to be obtained once. Multiple content categories can be identified using prompt words or fine-tuning models that identify different content categories in the task rules. This allows for the recognition of multiple content categories while maintaining a low cost for prompt words or models, reducing the cost of image content understanding while improving the ability and efficiency of image content understanding. Attached Figure Description

[0020] Figure 1a is a schematic diagram of the framework of an image content understanding platform provided in an exemplary embodiment of this application;

[0021] Figure 1b is a schematic diagram of the architecture of an image processing system provided in an exemplary embodiment of this application;

[0022] Figure 2a is a schematic diagram of how modules in an image content understanding platform based on Zero-Shot technology call each other to implement task rule configuration, according to an exemplary embodiment of this application.

[0023] Figure 2b is a schematic diagram illustrating how modules in an image content understanding platform based on Few-Shot technology call each other to implement task rule configuration, according to an exemplary embodiment of this application.

[0024] Figure 3 is a flowchart illustrating an image processing method provided in an exemplary embodiment of this application;

[0025] Figure 4 is a flowchart illustrating another image processing method provided in an exemplary embodiment of this application;

[0026] Figure 5a is a schematic diagram of accessing services in an image content understanding platform provided by an exemplary embodiment of this application;

[0027] Figure 5b is a schematic diagram of creating a content understanding task in an image content understanding platform according to an exemplary embodiment of this application;

[0028] Figure 6a is a schematic diagram of an image content understanding process provided in an exemplary embodiment of this application;

[0029] Figure 6b is a schematic diagram comparing the effects of existing mainstream content understanding and the content understanding of the present solution in an exemplary embodiment of this application;

[0030] Figure 7 is a flowchart illustrating another image processing method provided in an exemplary embodiment of this application;

[0031] Figure 8 is a flowchart illustrating another image processing method provided in an exemplary embodiment of this application;

[0032] Figure 9a is a schematic diagram of a business party building a sample library through an image content understanding platform, provided by an exemplary embodiment of this application;

[0033] Figure 9b is a schematic diagram illustrating the configuration of storage duration for business data in a sample library according to an exemplary embodiment of this application;

[0034] Figure 9c is a schematic diagram of configuring a data retrieval function for a sample library according to an exemplary embodiment of this application;

[0035] Figure 10 is a schematic diagram of a business data entry sample library provided in an exemplary embodiment of this application;

[0036] Figure 11 is a schematic diagram of the structure of a CLIP model provided in an exemplary embodiment of this application;

[0037] Figure 12a is a schematic diagram of a prompt word development process provided by an exemplary embodiment of this application;

[0038] Figure 12b is a schematic diagram of an interface for prompt word development provided in an exemplary embodiment of this application;

[0039] Figure 13 is a schematic diagram of a training process for model fine-tuning provided in an exemplary embodiment of this application;

[0040] Figure 14a is a schematic diagram of a model fine-tuning interface provided in an exemplary embodiment of this application;

[0041] Figure 14b is a schematic diagram of another model fine-tuning training process provided by an exemplary embodiment of this application;

[0042] Figure 15a is a schematic diagram of the interface for a task rule for creating a new content understanding task provided in an exemplary embodiment of this application;

[0043] Figure 15b is a schematic diagram of an interface for adjusting the original task rules of a content understanding task according to an exemplary embodiment of this application;

[0044] Figure 15c is a schematic diagram of an interface for adding new task conditions provided in an exemplary embodiment of this application;

[0045] Figure 16a is a schematic diagram of a rule preview interface provided in an exemplary embodiment of this application;

[0046] Figure 16b is a schematic diagram of the background process of a rule preview provided in an exemplary embodiment of this application;

[0047] Figure 17a is a schematic diagram of a rule evaluation interface provided in an exemplary embodiment of this application;

[0048] Figure 17b is a schematic diagram of the background process of rule evaluation provided in an exemplary embodiment of this application;

[0049] Figure 17c is a snapshot diagram of a rule evaluation provided in an exemplary embodiment of this application;

[0050] Figure 18 is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this application;

[0051] Figure 19 is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0052] This application provides an image processing solution, specifically an AI-assisted image content understanding solution. This solution relies on AI to understand the content of an image and identify its content category. For ease of understanding, some technical terms and related concepts involved in the image processing solution provided in this application are briefly introduced below; wherein:

[0053] (1) Content comprehension.

[0054] Content understanding refers to the process of understanding the content of multimedia resources. In this application embodiment, it mainly involves image content understanding, that is, the process of understanding the content of images. Image content understanding refers to the semantic understanding of the image content, specifically the process of comprehensively understanding the content category (or simply category) of the image objects in the image content, the relationships between image objects, and the scene depicted in the image content to interpret the image content. Here, the image objects in the image content refer to the main subject in the image; the content category of the image objects includes, but is not limited to, animals (such as people, cats, dogs, and fish), objects (such as cars, tables, and televisions), and landscapes (such as plains, mountains, and snow). By performing content understanding on images, the semantics and content category of the images can be effectively extracted, thereby helping to better complete downstream image-related tasks.

[0055] (2) Artificial intelligence.

[0056] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. The fundamental technologies of AI generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. The software technologies of AI mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0057] The image processing solutions provided in this application primarily relate to machine learning and deep learning within the field of artificial intelligence. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Deep learning is generally considered a type of machine learning. Machine learning uses simpler concepts such as predictive models, while deep learning uses artificial neural networks designed to mimic human thinking and learning. Machine learning can be viewed as a task whose goal is to enable machines (computers in a broad sense) to acquire human-like intelligence through learning. For example, if humans can identify targets of interest from images or videos, then computer programs (AlphaGo or AlphaGo Zero) are designed to master the ability to identify targets. Various methods can be used to implement machine learning tasks, such as neural networks, linear regression, decision trees, support vector machines, Bayesian classifiers, reinforcement learning, probabilistic graphical models, and clustering.

[0058] Furthermore, the embodiments of this application mainly involve feature extraction models and classification models under machine learning. Specifically: ① Feature extraction models, also known as image encoders, have the ability to extract image feature vectors; that is, feature extraction models refer to a type of neural network model capable of understanding images and generating image feature vectors. The image feature vector can be represented as an embedding, a low-dimensional vector that is easy for machine learning models to process and understand. By converting high-dimensional image data into low-dimensional vectors that capture key information in the image, it is more beneficial to perform subsequent calculations based on these low-dimensional vectors. Feature extraction models may include, but are not limited to: CLIP (Contrastive Language-Image Pre-training, a multimodal model), CLIP variants (such as WeCLIP), and other models capable of generating embeddings. ② Classification models have the ability to classify and recognize images; that is, classification models refer to a type of neural network model capable of dividing images into different content categories. For example, classification models include binary classification models, multi-class classification models, etc.; binary classification models can determine whether an image belongs to a certain content category, such as determining whether an email is spam; multi-class classification models can identify the probability of an image belonging to multiple content categories, such as the probability that an image belongs to a cat is 10%, the probability that it belongs to a dog is 50%, and the probability that it belongs to a bird is 40%, etc.

[0059] In practical applications, image content understanding faces challenges such as high barriers to entry, high costs, and low efficiency, hindering its large-scale application. For example, image cue word recognition requires developers with specialized algorithmic knowledge to create end-to-end models. This high technical barrier limits the ability of non-technical personnel (such as business personnel) without such expertise to achieve image content understanding. Furthermore, the development of traditional end-to-end models for cue word recognition is time-consuming, reducing the efficiency of image content understanding. Additionally, the number of content categories a single end-to-end model can recognize is limited. Enhancing the recognition capabilities across multiple content categories requires developing numerous end-to-end models, significantly increasing computational costs and drastically reducing the efficiency of image content understanding.

[0060] To improve the efficiency of image content understanding and reduce its threshold and cost, the image processing solution provided in this application embodiment supports business personnel in developing prompt words, fine-tuning models, and setting task rules. This allows image content understanding processing to be performed based on pre-set task rules and online data flow in specific business scenarios. The main contents of the image processing solution provided in this application embodiment include: task rule configuration based on prompt words and model fine-tuning, and using task rules to achieve image content understanding processing. Specifically: ① Task rule configuration based on prompt words and model fine-tuning includes: developing prompt words based on zero-shot, fine-tuning models based on few-shot, and configuring task rules based on prompt words and model fine-tuning. ② Using task rules to achieve image content understanding processing refers to the process of performing content understanding processing on the image to be processed based on pre-configured task rules in actual business scenarios to obtain the content category of the image. The following is a brief introduction to the two parts mentioned above:

[0061] (1) Task rule configuration based on prompt words and fine-tuning model.

[0062] Cue words are information that indicates a content category. Fine-tuning models are models that can identify one or more content categories, such as binary or multi-class classification models.

[0063] The process of configuring task rules based on prompt words and fine-tuned models can include: on the one hand, developing at least one prompt word using Zero-Shot technology with zero samples (i.e., no training data) and setting an appropriate prompt word threshold for each prompt word. On the other hand, fine-tuning or refining the classification model using Few-Shot technology with few samples (i.e., a small amount of training data) to obtain at least one fine-tuned model and setting an appropriate model confidence level for each fine-tuned model. Based on the aforementioned developed prompt words and fine-tuned models, business personnel customize task rules for content understanding tasks according to the specific task requirements of content understanding tasks in the business scenario; the content understanding task in the business scenario refers to specific downstream tasks related to images in the business scenario, including but not limited to image review tasks or image recommendation tasks. Specifically, business users can set the prompt words and corresponding prompt word thresholds as a task condition and configure this task condition into the task rules. Alternatively, they can set the model identifier (information that uniquely identifies the fine-tuned model, such as model number, model ID, etc.) and corresponding model confidence as a task condition and configure this task condition into the task rules. This allows for pre-configuration of task rules based on prompt words and fine-tuned models. In other words, the task rules include task conditions related to prompt words and / or task conditions related to fine-tuned models.

[0064] In addition, before fine-tuning the model and using it, or before configuring and deploying the task rules, it is also possible to evaluate the fine-tuned model or task rules. The evaluation aims to test the predictive performance of the fine-tuned model or preview the execution results of the task rules, so as to better ensure the effectiveness of the fine-tuned model or task rules.

[0065] (2) Use task rules to realize the content understanding and processing of images.

[0066] Based on the above implementation method (1), after pre-configuring the task rules for the content understanding task in the business scenario, the computer device can directly obtain the task rules for the content understanding task in response to the content understanding task in the business scenario. The content understanding task indicates that the image input from the business side is processed for content understanding, and the task rules for the content understanding task are used to indicate the processing rules adopted when processing the image for content understanding. The computer device will also obtain the image feature vector of the image. The image feature vector represents the image features of the image, that is, the key information in the image; the image feature vector can be extracted using a feature extraction model. In this way, the computer device can directly perform content understanding processing on the image feature vector of the image based on the task rules to obtain the content understanding result of the image. The content understanding result includes the content category to which the image belongs; specifically, according to each task condition included in the task rules, the image feature vector of the image is processed for content understanding to obtain the content understanding result of the image.

[0067] Therefore, when computer devices perform content understanding processing on image feature vectors according to task rules, they do so based on prompts or fine-tuning models to identify the content category of the image. On one hand, directly following task rules greatly simplifies the complexity of content understanding processing for images. On the other hand, for the same image, the process only requires obtaining the image feature vector once. Prompts or fine-tuning models from the task rules can then be used to identify multiple content categories. This allows for the identification of multiple content categories while maintaining relatively low prompt or model costs, reducing the cost of image content understanding while improving the ability and efficiency of image content comprehension.

[0068] The image processing solution provided in this application embodiment can be deployed in an image content understanding platform (or image-text understanding platform). This image content understanding platform can be understood as a platform integrating rich capabilities such as Zero-Shot cue word development, Few-Shot model fine-tuning, task rule configuration, evaluation functions, and content understanding functions. Specifically, the content understanding platform provided in this application embodiment integrates a complete solution for image content understanding. It not only extracts image feature vectors based on a feature extraction model but also supports Zero-Shot cue word development, debugging, and evaluation, as well as automatic Few-Shot model fine-tuning and evaluation, thereby achieving one-stop, low-cost, low-threshold, and agile iterative image content understanding. An exemplary framework diagram of the image content understanding platform can be seen in Figure 1a; as shown in Figure 1a, the image content understanding platform has rich platform capabilities, such as evaluation functions, Zero-Shot cue word development functions, Few-Shot model fine-tuning functions, rule configuration functions, rule deployment functions, and sample library capabilities. Furthermore, the image content understanding platform also has the ability to access internal models and storage space, as well as external systems or platforms, to help the image content understanding platform better support and realize various platform capabilities.

[0069] Regarding the image content understanding platform, it should be noted that: 1) This application embodiment supports the business side to directly implement the aforementioned series of functions (such as Zero-Shot cue word development, Few-Shot model fine-tuning, and task rule configuration) through the image content understanding platform. In this way, business personnel can directly develop Zero-Shot cue words, fine-tune Few-Shot models, and configure task rules through the image content understanding platform. They can also achieve one-stop, visual, and agile model / rule evaluation and rule deployment on the pages provided by the image content understanding platform. This eliminates the need for business personnel to possess professional algorithm knowledge, effectively lowering the threshold for implementing image content understanding and thus significantly reducing the development costs required for image content understanding.

[0070] 2) The image content understanding platform provided in this application embodiment supports deployment in computer devices as a plugin or application. Specifically: ① The image content understanding platform is deployed in the computer device as a plugin; plugins include system-level or application-level plugins. For example, if the image content understanding platform is deployed as a system-level plugin, any application running on the computer device can call the plugin to perform content understanding processing on images within that application. Similarly, if the image content understanding platform is deployed as an application-level plugin within an application running on the computer device, such as as a content understanding platform for the application, then after the computer device starts the application, the application can use the image content understanding platform to perform content understanding processing on images within the application. ② The image content understanding platform is deployed in the computer device as an application, meaning that the image content understanding platform performs content understanding processing on images as an independent application.

[0071] An application can refer to a computer program designed to perform one or more specific tasks. Classifying applications according to different dimensions (such as how they operate, their functions, etc.) can yield the types of the same application across different dimensions. For example, based on how they operate, applications may include, but are not limited to: clients installed on a terminal, small programs that can be used without downloading and installation (as subroutines of the client), and web (World Wide Web) applications opened through a browser. As another example, based on their functional type, applications may include, but are not limited to: IM (Instant Messaging) applications, content interaction applications, etc. This application does not limit the specific type of image content understanding platform; this is merely a description.

[0072] Furthermore, the image content understanding platform provided in this application embodiment, as a solution for image content understanding, can understand and identify the content category to which an image belongs; this makes the image processing solution provided in this application embodiment applicable to various business scenarios that require content understanding of images. Business scenarios may include, but are not limited to: ① Live video streaming scenario: Each frame of a live video is sent to the image content understanding platform as an image to be processed on the business side. The image content understanding platform can then perform content understanding on the video content of each frame of the live video. ② Audio and video scenario: Audio and video scenarios can be further subdivided into short video scenarios and long video scenarios, based on video duration; in video scenarios, the image content understanding platform can perform content understanding on each frame of a user-uploaded video to be published. ③ Driving scenario: Driving scenarios often require analysis of the vehicle's surrounding environment. The image content understanding platform performs content understanding on environmental images uploaded by the vehicle, contributing to safer driving. ④ Medical image analysis scenario: The image content understanding platform can perform content understanding on images in the medical field, helping doctors quickly classify and diagnose diseases. The above description is merely an exemplary product performance and business scenario given in the embodiments of this application, and does not limit the product performance and business scenario of the image processing solution provided in the embodiments of this application.

[0073] It should be noted that the image content understanding platform can simultaneously support image content understanding in one or more business scenarios. For example, the image content understanding platform can be deployed as a plugin in a client, allowing various business scenarios within that client to call the platform to achieve image content understanding, such as live video streaming, red envelope (digital cash gift) scenarios, and gaming scenarios. Furthermore, each business scenario can create independent content understanding tasks, and these tasks may be identical across different scenarios. A content understanding task created within a business scenario can be understood as a task that achieves or realizes content understanding; for example, content understanding tasks may include image review tasks and resource recommendation tasks.

[0074] For example, content understanding can be used as an image review task. In this case, after obtaining the content understanding result of an image, the computer device performs content review processing on the image based on the result, generating content review information that describes whether the image meets the requirements of the business scenario. For instance, if the scenario requirement for a video is that videos should not flaunt wealth, then after obtaining the content understanding result, the computer device will perform content review processing on the image based on the content category represented by the result. Specifically, it will determine whether the content category is flaunting wealth and generate content review information accordingly. If the content understanding result indicates that the image belongs to the category of money, the generated content review information could be: the image content is flaunting wealth; if the content understanding result indicates that the image belongs to the category of scenery, the generated content review information could be: the image content is not flaunting wealth.

[0075] For example, content understanding can be a resource recommendation task. In this case, after obtaining the content understanding result of an image, the computer device performs resource classification processing on the image based on this result, generating content classification information for the image. This content classification information describes the user group to which the image should be distributed when it is recommended / distributed as a resource. For instance, if the content understanding result indicates that the image belongs to the game category, then when the computer device performs resource classification processing on the image based on this result, and determines that the image belongs to the game category, the generated content classification information for the image can be expressed as: the image should be assigned to user groups whose interests are games.

[0076] It should be noted that the embodiments of this application do not limit the type and number of content understanding tasks created in the business scenario. For example, content understanding tasks may also include: content quality detection tasks, cross-modal retrieval tasks (modality refers to the form or data type of multimedia, such as text and images, which belong to two modalities), and image and text content classification tasks, etc.

[0077] For ease of understanding, the image processing scheme and image content understanding platform provided in this application embodiment will be briefly introduced below with reference to the system architecture shown in Figure 1b. As shown in Figure 1b, the system includes a first terminal 101, a second terminal 102, and a server 103. This application embodiment does not limit the number or naming of the first terminal 101, the second terminal 102, and the server 103. The devices in the system architecture are described below, including:

[0078] The first terminal 101 is a terminal device held / used by business personnel on the business side. Through the first terminal 101, business personnel control a series of platform capabilities on the image content understanding platform to achieve one-stop, low-threshold, and low-cost image content understanding. The second terminal 102 is a terminal device held / used by the user on the user side. Through the second terminal 102, the user can send the image to be processed to the image content understanding platform to achieve content understanding processing of the image. The terminal (such as the first terminal 101 or the second terminal 102) may include, but is not limited to, smartphones (such as smartphones running the Android system or smartphones running the Internetworking Operating System (IOS), tablet computers, portable personal computers, mobile Internet Devices (MIDs), in-vehicle devices, and head-mounted devices. This application embodiment does not limit the type of terminal device, but this is only a description of the specific type.

[0079] Server 103 is the server corresponding to the terminal (first terminal 101 or second terminal 102), used to interact with the terminal to provide computing and application service support. Server 103 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The first terminal 101, the second terminal 102, and server 103 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0080] The image processing scheme provided in this application embodiment is executed by a computer device, which can be a terminal and / or a server in the system architecture shown in Figure 1b; that is, the computer device can be executed by a terminal or a server, or jointly by a terminal and a server. For example, the computer device is server 103, which deploys an image content understanding platform, and the server 103 performs content understanding processing for images. As another example, the computer device includes a first terminal 101 and a server 103, and the platform capabilities of the image content understanding platform require the joint execution of the first terminal 101 and the server 103. The following describes the overall implementation process of the image processing scheme using the joint execution of the first terminal 101, the second terminal 102, and the server 103 as an example:

[0081] (1) The business side (including the aforementioned business personnel) configures the task rules on the image content understanding platform.

[0082] In specific implementation, the business party opens the service interface provided by the image content understanding platform through the first terminal 101. The business party can perform content understanding-related operations on this service interface, including but not limited to: selecting a sample library for Zero-Shot prompt word development, selecting a sample library for Few-Shot model fine-tuning, configuring task rules for content understanding tasks in the business scenario, and performing rule evaluation, etc. The task rules configured for the content understanding task can be stored in the form of key-value pairs. A key-value pair can be represented as a key-value pair, where the key is the key and the value is the value. For example, when storing task rules using key-value pairs, the key stores the rule identifier of the task rule, and the value stores the condition identifier of the task condition in the task rule. It should be noted that task rules can also be stored in other forms; this application embodiment does not limit the storage form of task rules, as long as it enables fast indexing and locating of task rules.

[0083] (2) The image content understanding platform performs content understanding processing on the online business data stream.

[0084] Depending on the task rules configured by the business user for content understanding tasks in the business scenario, the image content understanding platform employs different content understanding processing methods when receiving online data streams from the business user. For example, if the business user configures a task condition based on prompt words developed using Zero-Shot, the content understanding processing method includes performing content understanding processing on the image using prompt words and prompt word thresholds. As another example, if the business user configures a task condition based on a fine-tuning model of Few-Shot, the content understanding processing method includes performing content understanding processing on the image using the fine-tuned model and its model confidence. Furthermore, the content categories identified by each task condition in the task rule can be different or the same. For example, if at least two task conditions in the task rule identify the same content category, different task conditions can be used to make multiple judgments on the content category of the same image, thereby improving the accuracy of content judgment. On the other hand, if at least two task conditions in the task rule identify different content categories, different task conditions can be used to make multiple judgments on the content category of the same image, thereby quickly determining whether it belongs to multiple content categories and improving the speed and efficiency of content understanding.

[0085] The following section describes the specific process of task rule configuration and content understanding processing in conjunction with the various modules of the image content understanding platform. The image content understanding platform includes at least the following modules: a base model inference module, a feature library storage module, a content understanding module, a business subscription module, a prompt word debugging module (or tag debugging module), a sample library storage module, a rule module, a rule repository module, a tag caching module, a fine-tuning module, and a hosting module.

[0086] When the task conditions set in the task rules are based on prompt words developed using Zero-Shot, the general process of inter-module call between the prompt word debugging module, sample library storage module, rule module, rule repository, and tag cache module to configure the task rules can be seen in Figure 2a. As shown in Figure 2a: The business user sets prompt words on the service interface provided by the image content understanding platform. These prompt words are sent to the prompt word debugging module, which retrieves multiple sample images from the sample library storage module and segments these images, with each segment containing at least one sample image. The prompt word debugging module performs prompt word debugging on the prompt words and the sample images in each segment, obtaining matching information between each sample image and the prompt word in each segment. This matching information indicates whether the two are similar. The matching information is then displayed to the business user through the image content understanding platform, allowing the business user to set an appropriate prompt word threshold for the prompt word and use it as the prompt word. After the prompt word debugging is completed and the prompt words and prompt word thresholds are obtained, the business side can use the rule module in the image content understanding platform to set the prompt words and prompt word thresholds into the task conditions in the task rules; the task rules configured by the business side will be stored in the rule repository.

[0087] Furthermore, after the task rules for the content understanding task are configured, the general process of content understanding processing based on prompt words, implemented through the basis model inference module, feature library storage module, content understanding module, and business subscription module in the image content understanding platform, includes: When the image content understanding platform receives an image to be processed belonging to a business scenario, it submits the image to the corresponding content understanding task according to the content understanding purpose of the image. The computer device responds to the content understanding task, calls the feature extraction model (or basis model) through the basis model inference module to perform feature extraction processing on the image, obtains the image feature vector, and stores the image feature vector in the feature library storage module. When the feature library storage module receives the image feature vector, it triggers the content understanding module to perform content understanding processing on the image. The content understanding module performs content understanding processing on images as follows: It retrieves the task rules for the content understanding task from the rule storage module. If the task rules are cached in the tag caching module, the content understanding module directly retrieves them from the rule storage module. If the task rules are not cached in the tag caching module, the content understanding module retrieves them from the rule storage module through the tag caching module. By setting up a tag caching module in the image content understanding platform to cache used task rules, the workload of querying task rules from the rule storage module each time can be avoided, saving computing resources and improving the speed and efficiency of content understanding. After obtaining the content understanding result of the image, the content understanding module sends the result to the business subscription module so that the business subscription module can output the image content understanding result. Of course, if the content understanding task is a review task or a resource recommendation task, or other tasks, then after obtaining the image content understanding result, further processing (such as content review processing) is required based on the image content understanding task. The resulting content processing information (such as content review information) is then transmitted to the business subscription module.

[0088] When the task conditions set in the task rules are based on a fine-tuned model obtained by fine-tuning the Few-Shot model, the general process of mutual calls between the fine-tuning module and the managed module to configure the task rules can be seen in Figure 2b. As shown in Figure 2b: the business party initiates model fine-tuning on the service interface provided by the image content understanding platform. At this time, the image content understanding platform calls the fine-tuning module to fine-tune the content understanding model. This content understanding model is a model based on the feature extraction model. It also includes a classification model, which serves as the output layer of the content understanding model; therefore, the classification model is also called the Head layer or Head model. In other words, the feature extraction model serves as the base model, connected with the classification model to form the content understanding model; the input data of the content understanding model is the input data of the feature extraction model, the output data of the feature extraction model serves as the input data of the classification model, and the output data of the classification model is the output data of the content understanding model. During model fine-tuning, the model parameters of the feature extraction model are fixed, and the model parameters of the classification model are mainly fine-tuned to obtain the content understanding model after fine-tuning. Considering that the model parameters of the feature extraction model in the content understanding model are fixed, in this embodiment, model fine-tuning mainly focuses on the classification model, and the fine-tuned classification model is called the fine-tuned model. The fine-tuned model obtained by the fine-tuning module from the classification model can be deployed to the managed module, which can then call the fine-tuned model for inference and other purposes.

[0089] Furthermore, after deploying the fine-tuned model in the managed module, the general process of content understanding processing based on prompt words is implemented through the base model inference module, feature library storage module, content understanding module, and business subscription module in the image content understanding platform. This process is similar to the general process of content understanding processing based on prompt words shown in Figure 2a above. The following is a brief introduction to the general process of content understanding processing based on prompt words implemented through the base model inference module, feature library storage module, content understanding module, and business subscription module in the image content understanding platform. Specifically, the computer device responds to the content understanding task, calls the feature extraction model through the base model inference module to perform feature extraction processing on the image, obtains the image feature vector, and stores the image feature vector in the feature library storage module. When the feature library storage module receives the image feature vector, it triggers the content understanding module to perform content understanding processing on the image. The content understanding module performs content understanding processing on images as follows: The content understanding module passes the image feature vector to the managed module. The managed model then calls the fine-tuning model to perform content understanding processing on the image feature vector, obtaining the image's probability information. This probability information is returned to the content understanding module, which then obtains the image's content understanding result based on the probability information. The content understanding module then sends this result to the business subscription module, allowing it to output the image's content understanding result or perform other corresponding processing (such as content moderation) based on the result. The resulting content processing information is then transmitted back to the business subscription module.

[0090] In summary, the image content understanding platform and image processing solution provided in this application have at least the following advantages:

[0091] On the one hand, the embodiments of this application design an image content understanding platform based on a feature extraction model and a fine-tuned model. The image content understanding platform can perform image content understanding based on the business online data flow input from the business side and the task rules pre-configured by the business party, and realize the rapid identification of the content category to which the image belongs (that is, to label the image with the content category to which it belongs, with one label corresponding to one content category).

[0092] On the other hand, business users can directly develop Zero-Shot prompts using the prompt debugging module provided by the image content understanding platform, enabling prompt development and debugging directly on the platform. Simultaneously, they can fine-tune Few-Shot models using the fine-tuning module provided by the platform, training fine-tuned models with specific business samples from the sample library. Furthermore, by configuring task rules, the developed prompts and fine-tuned models can be directly deployed within the content understanding process. This not only provides a visual, one-stop interface for task rule configuration, evaluation / testing, and deployment, but also eliminates the need for specialized knowledge from business users, lowering the barrier to entry and reducing costs associated with content understanding.

[0093] On the other hand, it supports the transformation and encapsulation of the feature extraction capabilities of the feature extraction model, allowing the business side to directly use these capabilities. For the business side, it only needs to develop Zero-Shot prompts and fine-tune the Few-Shot model, using the developed prompts and fine-tuned model as downstream processing methods for the image feature vectors output by the feature extraction model (this method can achieve content category recognition). Thus, since the data in the business line data stream (such as the image to be processed) only passes through the feature extraction model once, incurring only one inference cost, compared to the high inference cost of multiple feature extractions required to infer different content categories in traditional technologies, the cost increase caused by adding prompts and fine-tuning models in this application embodiment is very small. Furthermore, with the relatively low cost of prompts and fine-tuning models, the flexibility in setting identifiable content categories is further improved.

[0094] It should be noted that (1) the system architecture shown in Figure 1b mentioned above in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems. That is to say, the schematic diagram of the system architecture shown in Figure 1b is an exemplary architecture schematic diagram; in actual applications, the number and types of devices included in the system may change, and the embodiments of this application do not limit the schematic diagram of the system architecture; for example, when the data flow on the business line is generated by the business side, the system may not include the second terminal 102.

[0095] (2) The data collection and processing in this application embodiment should strictly comply with the requirements of relevant laws and regulations. The acquisition of personal information must be based on the knowledge or consent of the individual (or have a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws and regulations and the authorization of the personal information subject. For example, when this application embodiment is applied to specific products or technologies, such as when performing content understanding on images, the permission or consent of the image publisher is required, and the collection, use and processing of relevant data (such as performing content understanding on images) must comply with the relevant laws, regulations and standards of the relevant regions.

[0096] As described above, the image processing method provided in this application mainly includes: performing content understanding processing on the business online data stream, and configuring task rules on the image content understanding platform. For ease of understanding, different embodiments will be used to describe the specific implementation process of these two parts in the following sections.

[0097] Please refer to Figure 3, which is a flowchart illustrating an image processing method provided in an exemplary embodiment of this application. This flowchart mainly illustrates the overall process of content understanding and processing for data streams on a business line. As shown in Figure 3, the general flow of the image processing method may include:

[0098] The task rules configured by the business side based on prompt words and / or fine-tuning models are pre-stored in the rule storage module of the image content understanding platform. Each task rule includes at least one task condition, and each task condition defines a content category and the corresponding content understanding processing method. Responding to the content understanding task in the business scenario, the computer device first performs feature extraction processing on the image to be processed in the business line data stream using the basis model deployed in the basis model inference module of the image content understanding platform, obtaining the image feature vector. The image feature vector is submitted to the content understanding module of the image content understanding platform. The computer device retrieves the task rules for the content understanding task from the rule storage module, and the content understanding module performs content understanding processing on the image feature vector according to the task conditions defined in the task rules, obtaining the content understanding result of the image.

[0099] Specifically, when the content understanding module performs content understanding processing on the image feature vector according to the task conditions, it does so according to the content understanding processing method defined in the task conditions. In this embodiment, the task conditions can be divided into cue word conditions and fine-tuning model conditions depending on the content understanding processing method defined in the task conditions. Optionally, if the task condition is a cue word condition, the content understanding processing method defined in the cue word condition includes: performing content understanding processing on the image through cue words and cue word thresholds; then, the general logic of the content understanding processing is: calculating the similarity between the image feature vector of the image and the word feature vector of the cue words, and determining whether the calculation result meets the relationship requirement of the cue word threshold. Optionally, if the task condition is a fine-tuning model condition, the content understanding processing method defined in this condition includes: performing content understanding processing on the image using the fine-tuning model and its model confidence. For example, assuming the task condition defines the use of fine-tuning models Head2 and Head4 for classification, the general logic of the content understanding processing is as follows: the content understanding module passes the image feature vector to the managed module, which then runs fine-tuning models Head2 and Head4 to classify the image feature vector, obtaining the probability information from the two fine-tuning models. The content understanding module then compares the probability information output by the two fine-tuning models with the model confidence defined in the task condition. It is worth noting that if both of the above task conditions are defined in the task rules, after obtaining the image's category result under each task condition, the content understanding result is also calculated based on the image's category result under each task condition.

[0100] Figure 3 above only provides a brief overview of the overall process of content understanding and processing of online business data streams. The following section, in conjunction with Figure 4, details the specific implementation process of content understanding and processing of online business data streams. Figure 4 shows a flowchart illustrating an image processing method provided in an exemplary embodiment of this application. The image processing method shown in Figure 4 mainly concerns the process of content understanding and processing of online business data streams. This image processing method can be executed by a computer device, which can be the server 103 shown in Figure 1b above. The image processing method may include, but is not limited to, steps S401-S403:

[0101] S401: Responding to content understanding tasks in business scenarios, obtain the task rules for content understanding tasks.

[0102] As described above, content understanding tasks in business scenarios are specific downstream tasks that perform content understanding processing on images within those scenarios. For example, a content understanding task might be an image review task. The purpose of image review is to perform content understanding processing on images within a business scenario to detect whether the image meets the scenario requirements, such as compliance with legal regulations. Furthermore, the task rules for content understanding tasks indicate the processing rules used when performing content understanding processing on images. These task rules are pre-configured by the business side, specifically by setting task conditions based on prompts developed using Zero-Shot prompts and / or fine-tuning models of Few-Shot models, and configuring task rules based on these conditions.

[0103] Based on this, when a computer device receives a business-line data stream from a business scenario, it submits the business data requiring content understanding processing within the stream to the specific content understanding task within that scenario. The business-line data stream refers to the data stream transmitted between devices within a business scenario; this stream is an ordered sequence of byte data with a start and end point, containing business data, such as images to be processed. When the computer device detects an image submitted for processing in a content understanding task, it responds to the task by retrieving the task rules. Specifically, it retrieves pre-configured task rules from the rule storage module of the image content understanding platform, enabling subsequent image processing based on these rules. Because the task rules are pre-configured and stored, the computer device can quickly obtain these rules during online content understanding, thereby improving the speed and efficiency of image content understanding processing and ensuring its real-time performance.

[0104] The image content understanding platform provided in this application embodiment can simultaneously support multiple business scenarios (or simply services). When each service accesses the image content understanding platform, it can first add its own content understanding task. Content understanding tasks between different services are isolated from each other to avoid data confusion risks caused by overlapping services, ensuring that content understanding tasks can be created independently in each business scenario. For example, an exemplary schematic diagram of a service accessing the image content understanding platform can be seen in Figure 5a. As shown in Figure 5a, the service interface 501 of the image content understanding platform includes a task component 502 corresponding to the content understanding task. When the task component 502 is selected, a service box 503 is displayed in the service interface 501. If the service wants to select or create a service, it can perform a trigger operation on the service box 503, at which time a service window 504 is output. This service window 504 includes the service data of previously created services and service creation options 5041. If the service wants to create a service, it performs a selection operation on the service creation option 5041. The computer device responds to the selection operation on the service creation option 5041 by outputting a service creation window 505. In this way, the business party can enter business data in the business creation window 505, and after confirming the entered business data, the computer equipment creates the information business in the image content understanding platform based on the received business data.

[0105] Similarly, each service accessed in the image content understanding platform can independently create content understanding tasks according to the service provider's task requirements, achieving isolation between content understanding tasks in different business scenarios within the image content understanding platform. For example, a schematic diagram illustrating how a service provider creates content understanding tasks for its business within the image content understanding platform can be seen in Figure 5b. As shown in Figure 5b, when the task component in the service interface of the image content understanding platform is selected, the service interface also displays task information for each content understanding task in the business scenario displayed in the business box. These content understanding tasks were created in a historical time. The service interface also displays a task creation option 506. If the service provider wants to create a new content understanding task in the business scenario, it can select this task creation option 506. In response to the selection of task creation option 506, the computer device outputs a task creation window 507. Thus, the service provider can input the task information for the content understanding task in the task creation window 507, and after confirming the input task information, the computer device creates the content understanding task in the business scenario within the image content understanding platform based on the received task information.

[0106] S402: Obtain the image feature vector of the image.

[0107] To facilitate better content category identification of images in the business data stream, this application embodiment supports the extraction of image feature vectors from the acquired image by the computer device. These image feature vectors characterize the image's features, containing key information about the image. Thus, content understanding based on image feature vectors can improve the accuracy of content understanding. As mentioned earlier, the image content understanding platform provided in this application embodiment deploys a content understanding model, which includes a basis model. Therefore, after acquiring an image, the computer device performs feature extraction processing on the image using the basis model deployed in the image content understanding platform to obtain the image feature vector.

[0108] S403: Perform content understanding processing on the image feature vectors according to the task rules to obtain the content understanding results of the image.

[0109] The task rules pre-configured by the business for content understanding tasks may include at least one task condition and the relationships between these task conditions. Specifically: ① The relationships between task conditions refer to the logical relationships between them. Specifically, when content understanding processing is performed on the image feature vector using each task condition in the task rules, the logical relationships between the category results of the image under each task condition are considered. These relationships include parallel or combined relationships. A parallel relationship corresponds to the logical relationship "OR," meaning that one category result is selected from the category results of the image under multiple task conditions as the basis for generating the content understanding result of the image. A combined relationship corresponds to the logical relationship "AND," meaning that multiple category results are selected from the category results of the image under multiple task conditions as the basis for generating the content understanding result of the image.

[0110] ② Each task condition can be understood as a sub-rule used for content understanding processing of the image feature vector of an image. Each task condition defines a content category and the corresponding content understanding processing method. The content category defined in the task condition indicates whether the image belongs to the corresponding content category when performing content understanding processing on the image feature vector of the image according to the task condition. The content understanding processing method corresponding to the content category defined in the task condition indicates the specific processing method used when performing content understanding processing on the image feature vector of the image according to the corresponding task condition. In the embodiments of this application, the content understanding processing methods mainly include the following two: performing content understanding processing on the image through prompt words and prompt word thresholds, and performing content understanding processing on the image through fine-tuning the model and the model confidence of the fine-tuning model. Of course, if there are other content understanding processing methods besides these two methods, the embodiments of this application also support the business party to configure them into the task rules as task conditions; for ease of explanation, the following embodiments will use the above two content understanding processing methods as examples for explanation, which is hereby described.

[0111] In specific implementation, after acquiring the image feature vector and the task rules for the content understanding task, the computer device performs content understanding processing on the image feature vector according to the prompt word conditions and fine-tuning model conditions corresponding to the two content understanding processing methods mentioned above. The reasoning process for obtaining the content understanding result of the image can be seen in Figure 6a. As shown in Figure 6a, the base model deployed in the image content understanding platform is called to perform feature extraction processing on the image, obtaining the image feature vector. According to the content understanding processing methods corresponding to each task condition defined in the task rules of the content understanding task, the image feature vector is processed to obtain the category result of the image under each task condition. The content understanding processing methods corresponding to each task condition defined in the task rules can be the same or different, and this is not limited.

[0112] Therefore, compared to traditional technologies (such as developing traditional end-to-end models or pre-trained large models as shown in Figure 6b), where each content category prediction requires an independent deployment of a model or module to infer the image feature vector once, the embodiments of this application share the image feature vector of the base model for each task condition during actual online content understanding. In other words, only the base model is needed to perform feature extraction processing on the image once, resulting in only one inference cost. In the embodiments of this application, the addition of Zero-Shot cue word development and Few-Shot model fine-tuning causes a small inference cost, thus avoiding a significant increase in cost due to the addition of cue words or fine-tuned models.

[0113] Figure 6a above only introduces the general process of content understanding processing. The following is a detailed implementation process of the computer device performing content understanding processing on the image feature vector of the image according to the task rules. This process may include steps s11 and s12:

[0114] s11: The computer device performs content understanding processing on the image feature vector according to each of the at least one task condition included in the task rules, specifically the content understanding processing method defined in each task condition, to obtain the category result of the image under each task condition. The category result is used to characterize whether the image belongs to the content category defined by the corresponding task condition, or whether the image does not belong to the content category defined by the corresponding task condition. In this embodiment, the specific implementation process of content understanding processing is described using prompt word conditions and fine-tuning model conditions as examples.

[0115] (1) The task conditions are cue word conditions. The content understanding processing methods defined in the cue word conditions include: performing content understanding processing on images through cue words and cue word thresholds. The cue words and cue word thresholds are pre-developed and debugged by the business side using Zero-Shot. Each cue word corresponds to a content category, and the form of the cue word can include, but is not limited to, text, images, or animations. The cue word threshold is a numerical value (such as a decimal or percentage) that serves as a standard for measuring whether an image belongs to the content category defined by the cue word conditions. Cue words have advantages such as simple data structure, strong scalability, and low computational complexity. This allows business personnel to develop a large number of cue word conditions according to their own task requirements, satisfying their personalized task condition needs without incurring significant costs.

[0116] When the task condition is a prompt word, the computer device acquires the word feature vector of the prompt word. This word feature vector is used to represent the semantic features of the prompt word. The word feature vector can be extracted by the base model and stored in the repository during the pre-execution of Zero-Shot prompt word development, or it can be extracted in real-time by the base model during online content understanding; there is no limitation on this. The computer device calculates the similarity between the image feature vector and the word feature vector to obtain a similarity result. The similarity result is used to represent the degree of similarity between the image and the prompt word. The similarity calculation can be obtained through cosine similarity, which measures the similarity between two feature vectors by calculating the cosine value of the angle between the two feature vectors in multidimensional space. The computer device compares the similarity result with the prompt word threshold to obtain the image's category result under the prompt word condition. If the similarity result is greater than or equal to the prompt word threshold, the image's category result under the prompt word condition is used to represent that the image belongs to the content category defined by the prompt word condition, and the image can be labeled with the corresponding prompt word. If the similarity result is less than the prompt word threshold, the image's category result under the prompt word condition is used to represent that the image does not belong to the content category defined by the prompt word condition, and the image does not need to be labeled with the corresponding prompt word.

[0117] (2) The task condition is a fine-tuning model condition. The content understanding processing method defined in the fine-tuning model condition includes: performing content understanding processing on images through the fine-tuning model and the model confidence of the fine-tuning model. The fine-tuning model and model confidence are obtained in advance by the business side through Few-Shot fine-tuning. A fine-tuning model is used to identify one or more content categories, and the model confidence is a numerical value (such as a decimal or percentage), which is the standard for measuring whether an image belongs to the content category identified by the fine-tuning model. As a machine learning model, the fine-tuning model can distinguish the content category of relatively complex images. As a supplement to the cue word condition, it can ensure that complex images in business scenarios can also be accurately distinguished in terms of content category, significantly improving the accuracy of content understanding.

[0118] When the task condition is fine-tuning the model, the computer device invokes the fine-tuning model to perform content understanding processing on the image feature vector, obtaining the image's probability information. This probability information represents the probability that the image belongs to the content category defined by the fine-tuning model (i.e., the content category that the fine-tuning model can recognize). The computer device then compares the image's probability information with the model confidence score to obtain the image's category result under the fine-tuning model condition. Specifically, if the probability information is greater than or equal to the model confidence score, the image's category result under the fine-tuning model condition represents that the image belongs to the content category defined by the fine-tuning model condition; if the probability information is less than the model confidence score, the image's category result under the fine-tuning model condition represents that the image does not belong to the content category defined by the fine-tuning model condition.

[0119] s12: The computer device processes the category results of the image under each task condition according to the association between the task conditions defined in the task rules, and generates the content understanding results of the image.

[0120] As mentioned earlier, the relationships between the various task conditions included in the task rules can be parallel or combined. If the relationships between the task conditions are parallel, when the computer device processes the category results of the image under each task condition, it compares the category results of the image under each task condition, aiming to select the content category with the highest probability information from the multiple content categories corresponding to multiple task conditions as the most likely content category to which the image belongs. By judging the content category of the image through multiple task conditions and determining the final content category based on the relationships between the task conditions, the accuracy of content category judgment for images is improved compared to using a single task condition, avoiding the decrease in content understanding accuracy caused by being limited by a single task condition. For example, the task rules include cue condition 1, cue condition 2, and fine-tuning model condition 3. Cue condition 1 corresponds to the content category "cat," cue condition 2 corresponds to the content category "dog," and fine-tuning model condition 3 corresponds to the content category "bird." If the image's category result under cue condition 1 is 20% with a cue threshold of 50%, the category result under cue condition 2 is 50% with a cue threshold of 30%, and the category result under fine-tuning model condition 3 is 60% with a model confidence of 50%, then the image's category result 1 under cue condition 1 is determined to be "not a cat," the image's category result 2 under cue condition 2 is determined to be "a dog," and the image's category result 3 under fine-tuning model condition 3 is determined to be "a bird." Then, when the association relationship is a parallel relationship, one category result from category result 2 and category result 3 is selected as the basis for generating the content understanding result. For example, if category result 2 is selected, the generated content understanding result might be "the image contains a dog."

[0121] If the relationship between task conditions is a combined use relationship, when the computer device processes the category results of the image under each task condition, it will compare the category results of the image under each task condition. The aim is to select from multiple content categories corresponding to multiple task conditions to characterize the category result that the image belongs to the content category defined by the corresponding task condition. The content category corresponding to this category result is taken as the content category to which the image belongs. For example, the task rules include cue condition 1, cue condition 2, and fine-tuning model condition 3. Cue condition 1 corresponds to the content category "cat," cue condition 2 corresponds to the content category "dog," and fine-tuning model condition 3 corresponds to the content category "bird." If the image's category result under cue condition 1 is 20% and the cue threshold is 50%, the category result under cue condition 2 is 50% and the cue threshold is 30%, and the category result under fine-tuning model condition 3 is 60% and the model confidence is 50%, then the image's category result 1 under cue condition 1 is determined to be "not a cat," the image's category result 2 under cue condition 2 is determined to be "a dog," and the image's category result 3 under fine-tuning model condition 3 is determined to be "a bird." Then, when the association relationship is a combined relationship, the computer device generates the image's content understanding result based on category result 2 and category result 3, such as "the image includes both dogs and birds."

[0122] In summary, this application embodiment supports the transformation and encapsulation of the feature extraction capabilities of the feature extraction model, enabling the business side to directly utilize these capabilities. Thus, in actual online content understanding processing, only one feature extraction process needs to be performed on the image using the feature extraction model, incurring only one inference cost. Compared to the high inference cost of traditional technologies requiring multiple feature extractions to infer different content categories, the cost increase caused by adding prompt words and fine-tuning the model in this application embodiment is minimal. Furthermore, with the relatively low cost of prompt words and model fine-tuning, the flexibility in setting identifiable content categories is further improved.

[0123] The embodiments shown in Figures 3 and 4 above mainly introduce the process of content understanding processing of online business data streams; the following, in conjunction with the embodiments in Figures 7 and 8, introduces the specific content of task rule configuration on the image content understanding platform.

[0124] Please refer to Figure 7, which is a flowchart illustrating an image processing method provided in an exemplary embodiment of this application; this flowchart mainly shows the overall process of configuring task rules on an image content understanding platform. As shown in Figure 7, the general flow of the image processing method may include:

[0125] Building a sample library. The sample library is a storage space within the image content understanding platform used to store business data from various business scenarios. Building a sample library specifically involves importing business data from these scenarios into the library, facilitating the direct use of the sample library for Zero-Shot prompt word development and Few-Shot model fine-tuning. For example, building a sample library includes: the business side creating a sample library through the image content understanding platform; then, after feature extraction processing of the business data from the online data stream, storing the feature vectors of the business data in the sample library, thus achieving data import into the sample library.

[0126] Cue word development and model fine-tuning. After building the sample library, business users can select samples from the image content understanding platform to develop cue words and fine-tune the model. Cue word development is based on data from the sample library and Zero-Shot technology, allowing for cue word development and debugging even without training samples. Model fine-tuning involves selecting a small amount of data from the sample library as training samples to fine-tune or adjust the classification model, enabling model fine-tuning with a limited number of samples. Task rule configuration. After developing cue words and fine-tuning the model based on the aforementioned steps, business users can configure task rules independently through the service interface provided by the image content understanding platform, tailored to their specific business needs, to meet their customized requirements for task rules.

[0127] Content understanding processing is performed on online business data streams based on task rules. After the task rules are pre-configured, when the computer device receives the online business data stream to be processed, it can directly retrieve the task rules for the content understanding task from the rule storage module to perform content understanding processing, which improves the speed and efficiency of content understanding to a certain extent.

[0128] Figure 7 above only provides a brief overview of the overall process of configuring task rules on the image content understanding platform. The specific implementation process of configuring task rules on the image content understanding platform will be described in detail below with reference to Figure 8. Figure 8 shows a flowchart of an image processing method provided by an exemplary embodiment of this application. The image processing method shown in Figure 8 mainly concerns prompt word development, model fine-tuning, and task rule configuration. This image processing method can be executed by a computer device, which may include the first terminal 101 and server 103 shown in Figure 1b above. The image processing method may include, but is not limited to, steps S801-S806:

[0129] S801: In response to the business party's sample library building operation in the image content understanding platform, create a sample library.

[0130] In this embodiment, each service accessed by the image content understanding platform corresponds to one or more sample libraries, which store service data and / or feature vectors of the corresponding service. In practical applications, service providers can create new sample libraries or modify existing sample libraries in their service scenarios through the image content understanding platform according to their needs. Specifically, the service provider performs a sample library building operation in the service interface provided by the image content understanding platform, and the computer device responds to this operation by creating a sample library. The sample library building operation can be a single user operation or a series of consecutive user operations, which is not limited in this embodiment.

[0131] For example, the exemplary process of a business party creating a sample library by performing a sample library construction operation through an image content understanding platform can be found in Figure 9a. As shown in Figure 9a, the service interface provided by the image content understanding platform includes a sample library component 901. When the sample library component 901 is selected, a list of sample libraries corresponding to the business scenario is displayed on the service interface. This list includes library information (such as library identifier, library name, and entry method) of sample libraries created for the business scenario in the past. The service interface also includes a library creation option 902. When the library creation option 902 is triggered, it indicates that the business party wants to create a new sample library for the business scenario. At this time, the computer device responds to the trigger operation of the library creation option 902 and outputs a library creation window 903. The library creation window 903 is used to configure information for the sample library to be created. The information that can be configured in the library creation window 903 includes, but is not limited to, library identifier, library name, entry method, data storage period, and data retrieval information.

[0132] In detail, the database creation window 903 displays configuration areas corresponding to each type of information to be configured, such as configuration area 9031 for the import method, configuration area 9032 for the data storage period, and configuration area 9033 for data retrieval. When configuration area 9031 is triggered, one or more candidate import options are output, each corresponding to an import method. The import method refers to the way business data is imported into the sample database, including but not limited to: importing business data into the sample database through a backend interface (such as importing data through an interface provided by an image content understanding platform); importing business data in a certain data format into the sample database (such as importing business data in CSV format, but not limited to CSV); and importing business data from a storage database (such as a TDW (Tencent Distributed Data Warehouse), Kafka, or HDFS (Hadoop Distributed File System) database), etc. The business can select a target entry option from one or more candidate entry options, and the computer device will set the entry method corresponding to the target entry option to the entry method of the created sample library.

[0133] As shown in Figure 9b, when configuration area 9032 is triggered, one or more candidate duration options are output. Each candidate duration option corresponds to a storage duration, which refers to the maximum storage time of business data in the sample database (such as permanent, 7 days, one month, or one year). When the storage time of business data in the sample database reaches the storage duration, this part of the business data can be understood as expired data, and expired data can be deleted to save storage space in the sample database. The business party can select a target duration option from one or more candidate duration options according to its own data storage needs. At this time, the computer device sets the storage duration corresponding to the target duration option to the maximum storage time of business data in the sample database.

[0134] As shown in Figure 9c, the configuration area 9033 includes a data retrieval switch. When the data retrieval switch is turned on, one or more input fields related to data retrieval are displayed; when the data retrieval switch is turned off, one or more input fields related to data retrieval are hidden. With the data retrieval switch on, business users can input information in the input fields of the configuration area 9033 to configure information related to data retrieval. Configuring data retrieval for the sample library aims to determine whether the sample library supports online retrieval. Online retrieval refers to retrieving content similar to the input business data from the sample library; for example, users can use the online retrieval function in the image content understanding platform to retrieve desired sample images from the sample library. In this embodiment, it is supported to set up a vector library, or an index library and a storage library for online retrieval (or online search) in the configuration area 9033. Specifically: ① The vector library is used to store sample data (such as sample feature vectors) of sample images; the vector library can be the SimOL library or other retrieval libraries with vector retrieval capabilities; when configuring the vector library in the configuration area 9033, the address and path of the vector library need to be configured. ② The index library stores the library identifier and the sample image identifier within the library in key-value pairs, allowing indexing of the corresponding sample image identifier based on the library identifier. The storage library specifically stores the sample image identifier and related sample information (such as sample image metadata) in key-value pairs, enabling quick indexing of the corresponding sample image's related information based on the sample identifier. The index library can be a TableKV library, and the storage library can be a TTLKV library. This embodiment supports configuring the storage library path and address in configuration area 9033. It should be noted that when setting up data retrieval functionality for the sample library, the business can choose an existing vector library as the corresponding vector library or automatically apply for a new vector library; there are no limitations on this.

[0135] It should be noted that: ① The image content understanding platform provided in this application embodiment can achieve data retrieval through the vector library, index library, and storage library configured in Figure 9c; however, it should be understood that this application embodiment also supports data retrieval in the image content understanding platform through other retrieval methods, and this application embodiment does not limit the specific implementation process of data retrieval in the image content understanding platform. ② The sample library mentioned in this application embodiment can be, in addition to the aforementioned SimOL library, or TableKV library and TTLKV library, other databases; this application embodiment does not limit the type of sample library.

[0136] Based on the sample library construction operations shown in Figures 9a, 9b, and 9c, after the business party completes the information configuration for the sample library, the computer equipment can create the sample library according to the information configured by the business party and import business data into it. The process of the computer equipment importing business data into the sample library generally includes: the computer equipment acquiring a data stream, which refers to a data stream composed of multiple business data related to the business scenario, such as the aforementioned business line data stream; this data stream includes multiple business data—sample images, which may or may not be tagged. The computer equipment performs image analysis processing on each of the multiple sample images to obtain sample data for each sample image. The image analysis processing performed on the sample images can include various types of analysis processing, such as feature extraction processing, in which the sample data includes at least the sample feature vector of the sample image; or image numbering processing, in which the sample data includes a sample identifier that can uniquely identify the sample image in the sample library. The computer equipment stores the sample data of each sample image into the sample library, thus realizing the import of business data into the sample library.

[0137] In detail, based on the description shown in Figure 9c above, when setting up data retrieval information for the sample library, the business can achieve online retrieval of the sample library through a vector library, or an index library and a repository. Depending on these two retrieval methods, the way the computer device stores the sample data of each sample image in the sample library differs. The process of business data entering the sample library, specifically the index library and repository, or the vector library, is described below with reference to Figure 10. As shown in Figure 10, after the computer device obtains the data stream, it can choose to store the sample images in the data stream in Cloud Object Storage (COS). Specifically, it can store the binary data of the sample images in the object storage for easy access at any time. The computer device also performs image analysis processing on each sample image in the data stream to obtain sample data for each sample image; for example, it can use the basis model inference module in the image content understanding platform to perform feature extraction processing on each sample image in the data stream to obtain the sample feature vector of each sample image. The computer device can then store the sample images and the sample data of each sample image in the sample library. Specifically: If the business unit configures a vector library for the sample database, the computer device stores the sample feature vector of each sample image in the vector library. This allows for direct retrieval and use of the sample feature vector of the sample image from this vector library, eliminating the need for repeated feature extraction of the sample image and saving steps in online content understanding, thereby improving online content understanding efficiency. If the business unit configures an index library and a repository library for the sample database, the computer device stores the sample identifier of each sample image and the library identifier of the sample database as data pairs in the index library. Furthermore, it stores each sample image, its sample identifier, and related sample information in the repository library, including the sample feature vector of the sample image. By storing sample data in the repository and index library, the native data indexing relationship between the two libraries facilitates rapid data retrieval and utilization during content understanding, improving content understanding efficiency. Of course, if the business side configures a vector library, as well as an index library and a storage library for the sample library, then the computer equipment will perform data entry in both of the above methods. The advantage of these two methods is that, during the subsequent content understanding process, different data entry methods can be dynamically selected for data reading and writing based on the actual hardware or software resource requirements, thereby improving the flexibility of data reading and writing in the sample library.

[0138] S802: Based on the sample library, the prompt words are debugged to obtain prompt words and prompt word thresholds, and based on the sample library, the classification model is fine-tuned to obtain the fine-tuned model and model confidence.

[0139] In this application embodiment, two content understanding processing methods for images are provided: one method achieves content understanding processing through prompt words and prompt word thresholds, and the other method achieves content understanding processing through fine-tuning models and model confidence. The prompt words and prompt word thresholds can be obtained in advance by the business party through prompt word development / debugging using an image content understanding platform. Similarly, the fine-tuning model and model confidence are also obtained in advance by the business party through model fine-tuning / fine-tuning using an image content understanding platform. The business party independently develops the prompt words and fine-tuning models according to its own business requirements, and customizes the prompt word thresholds and model confidence, thereby improving the matching between the content understanding processing method and the business party's needs. The specific implementation process of prompt word development and model fine-tuning is described below:

[0140] (1) Development of prompt words.

[0141] This application primarily utilizes Zero-Shot technology to develop zero-shot cue words. Zero-Shot technology is a machine learning method that allows the model to handle categories never seen in the training data, enabling it to handle new tasks well without additional training and fine-tuning. The "model" here refers to a feature extraction model suitable for Zero-Shot learning. In this application, the CLIP model, a multimodal model suitable for Zero-Shot learning, is used as an example. In practical applications, other models can also be used for developing cue words using Zero-Shot technology, which is explained here.

[0142] The CLIP model's structure is illustrated in Figure 11. As shown, the CLIP model includes a text encoder and an image encoder. The logic of CLIP for zero-shot classification involves the text encoder processing the input modality—text—to obtain its text feature vector, while the image encoder processes the input modality—image—to obtain its image feature vector. The image feature vector and the text feature vector are then compared to obtain a similarity result representing the degree of similarity between the image and text. This similarity result is compared with the CLIP model's confidence score to determine whether the image matches the text. If the similarity result is greater than the CLIP model's confidence score, the image matches the text; otherwise, the image does not match the text.

[0143] To facilitate understanding, taking a business-side sample database configured with an index and a repository as an example, this section introduces the specific process of developing prompt words using the CLIP model-based classification and recognition logic; a schematic diagram of this process can be found in Figure 12a. As shown in Figure 12a:

[0144] 1) The business party inputs prompt words through the platform front-end of the image content understanding platform (such as the second terminal 102) so that the computer device can obtain the prompt words set by the business party in the image content understanding platform. A schematic diagram of the business party setting prompt words in the image content understanding platform is shown in Figure 12b. A tag debugging component 1201 is displayed on the service interface of the image content understanding platform. When the tag debugging component 1201 is selected, a prompt word input area 1202 is also displayed on the service interface. The business party can input prompt words in this prompt word input area 1202, at which point the computer device obtains the prompt words. The business party can also select a sample library for this prompt word development in the service interface. For example, the service interface also includes a sample library input area 1203, where the business party inputs the library identifier of the sample library to select the sample library.

[0145] 2) The computer device invokes the prompt word development module in the image content understanding platform to perform prompt word development processing based on the sample library and prompt words selected by the business party. Specifically, the prompt word development process includes: the prompt word development module inputs the prompt word into the CLIP model; the text encoding module in the CLIP model performs feature extraction processing on the prompt word to obtain the text feature vector of the prompt word—the word feature vector. The prompt word development module also calculates the matching information between the word feature vector and each sample image in the sample library selected by the business party, based on the word feature vector. The sample library here may include, but is not limited to: an index library (such as a TableKV library) and a storage library (such as a TTLKV library) configured by the business party for the sample library, or a vector library (such as a SimOL library). In this embodiment, using a vector library for prompt word development can accelerate the speed and efficiency of prompt word development. When the sample library includes an index library and a storage library, the prompt word development module first obtains the sample identifier of the sample image from the index library based on the library identifier of the sample library, and then obtains the sample image from the storage library based on the sample identifier. The prompt word development module inputs each sample image into the CLIP model. The image encoding module in the CLIP model performs feature extraction processing on each sample image to obtain the sample feature vector of each sample image (if the sample feature vector is stored in the repository, it is directly obtained). The prompt word development module then calculates the matching information between the word feature vector and the sample feature vector of each sample image; the higher the matching information between the word feature vector and the sample image, the higher the similarity between the sample image and the prompt word.

[0146] 3) The computer device can set an appropriate prompt word threshold based on the word feature vector of the prompt word and the matching information between the sample images in the sample library. Optionally, the image content understanding platform can output the matching information between the word feature vector of the prompt word and the sample feature vector of each sample image in the sample library in the service interface. Referring again to Figure 12b, after the business party sets the information required for this prompt word development (such as the prompt word and the library identifier of the sample library) in the service interface and clicks submit, the computer device displays the calculated word feature vector of the prompt word and the matching information between each sample image in the sample library in the service interface; for example, it displays sample image 1204, and displays the matching information 1205 (e.g., 0.647) between sample image 1204 and the word feature vector of the prompt word in an adjacent position. It also supports displaying other statistical information. In this case, the computer device can receive the prompt word threshold set by the business party based on the matching information. Optionally, after calculating the word feature vector of the prompt word and the matching information between each sample image in the sample library, the image content understanding platform can also automatically determine a suitable prompt word threshold based on the numerical distribution of each matching information.

[0147] Therefore, this application embodiment supports business users in understanding the platform's capabilities based on image content, enabling them to develop personalized zero-sample prompts without requiring professional algorithm knowledge, thus lowering the threshold for tag debugging. Furthermore, the matching information can be directly presented in the service interface, and this information visualization method helps business users grasp the details of tag debugging.

[0148] (2) Model fine-tuning.

[0149] This application primarily utilizes the Few-Shot technique to fine-tune a model with limited data. The goal of the Few-Shot technique is to train a classification model using a small number of labeled sample images (e.g., dozens of sample images) to obtain a fine-tuned model; that is, the classification model is the model before fine-tuning, and the fine-tuned model is the model after fine-tuning. The advantage of the Few-Shot technique is that it allows for the rapid construction and deployment of classification models in situations where data is scarce, thereby accelerating business deployment.

[0150] Specifically, it supports adding a classification model on top of a base model (such as the aforementioned feature extraction models, such as the WeCLIP model or CLIP model). The overall result of adding the classification model to the base model is the aforementioned content understanding model. In other words, the classification model to be fine-tuned is added to the output side of the base model, so that the image feature vector output by the base model serves as the input data for the classification model. The classification model classifies and identifies the image feature vector output by the base model to obtain probability information, which is used to characterize whether the image belongs to the label of the classification model.

[0151] In this embodiment of the application, model fine-tuning primarily focuses on adjusting the model parameters of the classification model within the content understanding model during the model training phase. For example, the training phase process of model fine-tuning can be seen in Figure 13, which includes:

[0152] 1) Sample Retrieval. Sample retrieval aims to select multiple target sample images from a sample library corresponding to the business scenario to construct a training dataset. The target sample images are the sample images in the sample library used to fine-tune the model. The classification model to be fine-tuned and the source of the samples can be customized by the business side through the image content understanding platform.

[0153] As shown in Figure 14a, the service interface of the image content understanding platform displays a model fine-tuning component 1401. When the model fine-tuning component 1401 is selected, the service interface displays one or more fine-tuning models corresponding to the business scenario and fine-tuning options 1402. In response to the business policy's triggering operation on the fine-tuning option 1402, the computer device displays a fine-tuning settings interface 1403. In this interface, the business user can select the classification model to be fine-tuned, the sample library from which the target sample images originate, the number of samples, and the labeling group (e.g., the working group that needs to manually label the target sample images selected from the sample library). After the business user sets the information in the fine-tuning settings interface 1403, the computer device will automatically select multiple target sample images from the corresponding sample library based on the set information. Specifically, it can use the Zero-Shot capability to recall an appropriate number of target sample images from the sample library. The recalled target sample images can be automatically saved to a new sample library for direct use later.

[0154] After recalling multiple target sample images using the Zero-Shot capability, an appropriate number of target sample images can be selected from these images for model fine-tuning based on an image selection strategy. Image selection strategies may include, but are not limited to: a hierarchical extraction strategy, such as sequentially extracting the first to tenth, 100th to 110th, and 200th to 210th target sample images from the 300 images if only 30 images are needed for model training; or a uniform extraction strategy, such as extracting one target sample image every 10 images from the 300 images for model fine-tuning. Of course, the specific strategy for image selection in this embodiment is not limited, but is described herein.

[0155] 2) Labeling.

[0156] As mentioned earlier regarding the construction of the sample database, some sample images imported into the database may carry labels. Therefore, if a second sample image with labels exists among the multiple target sample images retrieved, this second sample image does not need to be labeled, saving time spent on labeling and improving model fine-tuning efficiency. If a first sample image without labels exists among the multiple target sample images retrieved, this first sample image will be sent to a labeling group, where staff members will manually label it. The computer will then receive the labels added by the staff. Alternatively, the labeling of the first sample image can be achieved by calling a model with labeling capabilities. Based on this, the computer can obtain manually labeled first sample images and second sample images with their own labels. Thus, a training dataset can be constructed based on the labeled second sample images and the labeled first sample images from multiple sample images.

[0157] 3) Fine-tune the classification model based on the training dataset to obtain the fine-tuned model.

[0158] Figure 14b illustrates the process of fine-tuning the classification model using the training dataset. In one round of model training, target sample images from the training dataset are input into the content understanding model. The feature extraction model within this model (such as the image encoder WeCLIP) extracts features from the target sample images, obtaining their feature vectors. These feature vectors are then input into the classification model, which classifies the target sample images based on them, yielding probability information that represents the probability of the image belonging to a specific content category. The difference between the probability information and the label of the target sample image is calculated using the loss function of the content understanding model. This difference is then used to optimize the model parameters of the classification model within the content understanding model, resulting in the classification model after one round of training. The loss function can be the Binary Cross-Entropy (BCE) loss function, which is used to optimize the classification model using stochastic gradient descent. By training the classification model multiple times using this method, a well-performing fine-tuned model can be obtained.

[0159] Therefore, the embodiments of this application employ Few-Shot technology to fine-tune the classification model. Compared to developing a large end-to-end model, this reduces the workload and shortens the model training time, thereby improving content understanding efficiency. Furthermore, model fine-tuning is performed directly through an image content understanding platform, and the fine-tuned model is directly deployed on the platform to participate in online content understanding, lowering the barrier to model fine-tuning.

[0160] S803: Responds to rule configuration operations and creates task rules for content understanding tasks in business scenarios.

[0161] After the business side develops prompt words and fine-tunes the model on the image content understanding platform based on the aforementioned steps, it can set task rules for content understanding tasks in its business scenarios through the image content understanding platform. In this way, during online content understanding, the pre-configured task rules can be directly invoked to perform content understanding processing on the images indicated by the content understanding task. On the one hand, the business side can customize the task rules for content understanding tasks to meet its personalized image understanding needs in its business scenarios; on the other hand, pre-configuring the task rules effectively improves the efficiency of online content understanding.

[0162] The process of configuring task rules for content understanding tasks in an image content understanding platform, performed by the business party, can include: displaying a rule configuration interface for the content understanding task on the image content understanding platform, which includes a configuration area corresponding to the task rules. The business party can then set condition information for at least one task condition that constitutes a task rule in the configuration area, and the computer device receives this condition information. The condition information includes: the content understanding processing method (such as prompt words or fine-tuning models) and corresponding threshold information (such as prompt word thresholds or model confidence). If the business party submits the configured condition information in the rule configuration interface, the computer device responds to the submission operation by generating the task rules for the content understanding task, thereby realizing the creation of the task rules for the content understanding task.

[0163] Depending on the method used to trigger the display of the rule configuration interface, the rule configuration interface can be an interface for creating task rules for new content understanding tasks, or it can be an interface for existing task rules for content understanding tasks. For example, a schematic diagram of a rule configuration interface for creating task rules for new content understanding tasks can be seen in Figure 15a. As shown in Figure 15a, a rule component 1501 is displayed in the service interface of the image content understanding platform. When the rule component 1501 is selected, a rule creation option 1502 is displayed in the service interface. When the rule creation option 1502 is triggered, it indicates that the business party has a need to create new task rules for the content understanding task, and then the rule configuration interface 1503 for the content understanding task is displayed. This rule configuration interface 1503 includes multiple configuration areas for setting condition information for task conditions. For example, the rule configuration interface 1503 includes: a configuration area 1504 for the content understanding processing method of task condition 1, which is used to define the content understanding processing method of task condition 1, specifically configuring pre-developed prompt words or fine-tuned models; correspondingly, a configuration area 1505 for the threshold information corresponding to the content understanding processing method defined in task condition 1, which is used to define the prompt word threshold corresponding to the prompt word or the model confidence corresponding to the fine-tuned model.

[0164] For example, Figure 15b illustrates the rule configuration interface as a diagram of the original task rules for the content understanding task. As shown in Figure 15b, when the rule component 1501 in the service interface of the image content understanding platform is selected, the service interface also displays at least one task rule for the content understanding task. These task rules are configured historically, such as task rule 1506, task rule 1507, etc. If the business wants to reset a historically created task rule, it can perform a trigger operation on the task rule, at which time the rule configuration interface of the historically created task rule is displayed. For example, when task rule 1506 is triggered, the rule configuration interface of task rule 1506 is displayed, which shows at least one task condition included in task rule 1506 itself. The business can reset the task conditions, etc.

[0165] Based on the above rule configuration description, it is worth noting that: ① Regardless of whether the business party resets the original task rules for the content understanding task or creates new task rules for the content understanding task, this application embodiment supports adding task conditions in the rule configuration interface. As shown in Figure 15c, the rule configuration interface includes a new condition option 1508; when this new condition option 1508 is triggered, a new configuration area corresponding to task condition 2 is added to the rule configuration interface. The way the business party configures the condition information in the configuration area corresponding to task condition 2 is the same as the way the business party configures the condition information in the configuration area 1504 corresponding to task condition 1, and will not be elaborated here. By allowing the business party to customize the number and condition information of task conditions in the task rules according to its own task processing needs, the personalized rule configuration needs of the business party are met.

[0166] ② This application embodiment also supports multi-version management of task rules for content understanding tasks. Multi-version management means that the same task rule can have multiple version numbers. A task rule corresponding to a version number is reset once, and the incrementing of the version number indicates that the task rule has been reset multiple times. Based on this, if the business party completes the setting of task conditions in the rule configuration interface of the content understanding task, the computer device will not only generate task rules based on the condition information, but also generate a rule version number for the task rule. If the task rule generated based on the task conditions is a newly added task rule, then the rule version identifier of the task rule is the initial version number; if the task rule generated based on the task conditions is an update / reset of the original task rule of the content understanding task, then the rule version identifier of the task rule is generated based on the historical version number of the task rule. For example, if the historical version number is 1, then the historical version number is incremented to obtain the rule version identifier as 2. By managing task rules in multiple versions, it is convenient for the business party to easily backtrack on the task rules. For example, if the business party wants to use the task rule corresponding to the historical version, it can quickly backtrack and call the task rule corresponding to the historical version, improving the flexibility of task rule management.

[0167] Furthermore, this application embodiment also supports rule previewing for content understanding tasks. Rule previewing refers to running the task rules once or multiple times using sample images from the sample library before the task rules are officially launched. This allows for reasonable debugging of the task rules based on the preview results, helping business users understand the rule configuration effects.

[0168] This application embodiment supports business parties actively initiating rule previews, or automatic rule previews after task rules are created. The following describes the rule preview process using a business party actively initiating a rule preview as an example. Specifically, the rule configuration interface includes a sample library selection area and a rule preview component. The computer receives the library identifier selected by the business party for the task rule in the sample library selection area of ​​the rule configuration interface; this library identifier corresponds to a sample library. When the rule preview component is triggered, it indicates that the business party wants to preview the execution effect of the task rules configured in the rule configuration interface. The computer device responds to the trigger operation on the rule preview component, and based on the task rules created using the condition information set on the rule configuration interface, performs content understanding processing on each sample image in the sample library corresponding to the library identifier set in the sample library selection area, obtaining the content understanding result of each sample image. For example, the content understanding result of the sample image indicates that the sample image hits a label, or that the sample image does not hit a label. This application embodiment also supports outputting the content understanding results of each sample image to the business side; specifically, the content understanding results of each sample image are displayed in the rule configuration interface, such as displaying sample images whose content understanding results indicate that the tags are matched, and sample images whose content understanding results indicate that the tags are not matched, so as to facilitate the business side to preview.

[0169] For example, a schematic diagram of the rule preview interface implemented by the business party in the rule configuration interface can be seen in Figure 16a. As shown in Figure 16a, a sample library selection area 1601 is displayed in the rule configuration interface. The business party can set a library identifier in this sample library selection area 1601. The sample library corresponding to the library identifier serves as the source of the sample images used in this rule preview. A rule preview component 1602 is also displayed in the rule configuration interface. When the rule preview component 1602 is selected, the computer device creates a task rule based on the task condition information set in the rule configuration interface and performs content understanding processing on the sample library corresponding to the library identifier to obtain the content understanding results of each sample image in the sample library. The content understanding results of each sample image are then output in the rule configuration interface. As shown in Figure 16a, the hit area 1603 in the rule configuration area displays sample images whose content understanding results indicate that the tags have been hit, while the hit area 1604 in the rule configuration area displays sample images whose content understanding results indicate that the tags have not been hit.

[0170] For example, a schematic diagram of the background process for rule preview in a computer device (or image content understanding platform) can be seen in Figure 16b. As shown in Figure 16b, the business party sets up the task rules through the rule configuration interface displayed on the front end of the image content understanding platform, selects a sample library, and then calls the rule preview module of the image content understanding platform to send the task rules and the library identifier of the sample library. Assuming that the business party has pre-configured an index library TableKV and a storage library TTLKV for the sample library, the rule preview module interacts with the index library TableKV and the storage library TTLKV to obtain the sample feature vectors of each sample image in the sample library. The rule preview module also performs content understanding processing on the sample feature vectors of each sample image according to the content understanding processing method defined in the task conditions of the task rules, and obtains the content understanding results of each sample image. The rule preview module returns the content understanding results of each sample image to the platform front end for display.

[0171] Furthermore, this application embodiment also supports rule evaluation for content understanding tasks. Rule evaluation refers to running the task rules once or multiple times using sample images from a sample library before the task rules are officially launched, and then evaluating the results based on the rule preview. Designing a rule evaluation function in the image content understanding platform helps business users test the effectiveness of task rules before their official launch, thereby helping them configure task rules that better meet their business needs.

[0172] Similar to rule preview, this application embodiment supports business parties actively initiating rule evaluation, or automatic rule evaluation after task rule creation. The following describes the rule evaluation process using business parties actively initiating rule evaluation as an example. Specifically, the rule configuration interface also includes a sample library selection area and a rule evaluation component. The computer receives the library identifier selected by the business party for the task rule in the sample library selection area of ​​the rule configuration interface; this library identifier corresponds to a sample library. When the rule evaluation component is triggered, it indicates that the business party wants to evaluate the execution effect of the task rule configured in the rule configuration interface. The computer device responds to the trigger operation on the rule evaluation component by performing content understanding processing on each sample image in the sample library corresponding to the library identifier set in the sample library selection area, based on the task rule created according to the condition information of the task conditions set on the rule configuration interface, to obtain the content understanding results of each sample image. The computer device also performs evaluation statistical processing on the content understanding results of each sample image to obtain the evaluation results for the task rule. These evaluation results may include, but are not limited to, the recall rate and precision rate of sample images that hit the label in the sample library retrieved based on the task rule. Computer devices output evaluation results for task rules through an image content understanding platform, so that business users can intuitively perceive the execution effect of the set task rules. If the execution effect is not good, optimizing the task rules can improve the accuracy of subsequent image understanding based on task rules and improve the quality of image content understanding.

[0173] For example, a schematic diagram of the rule evaluation interface implemented by the business party in the rule configuration interface can be seen in Figure 17a. As shown in Figure 17a, a sample library selection area is displayed in the rule configuration interface. The business party can set a library identifier in this sample library selection area. The sample library corresponding to the library identifier serves as the source of sample images used in this rule preview. The rule configuration interface also displays a rule evaluation component 1701. When the rule evaluation component 1701 is selected, the computer device creates task rules based on the task conditions set in the rule configuration interface and performs content understanding processing on the sample library corresponding to the library identifier to obtain the content understanding results of each sample image in the sample library. The computer device performs evaluation and statistical processing on the content understanding results of each sample image, and the obtained evaluation results are output through the image content understanding platform. As shown in Figure 17a, the evaluation results can be displayed in the rule configuration interface or in a floating window, which is not limited.

[0174] For example, a schematic diagram of the background process of rule evaluation implemented by the computer device (or image content understanding platform) can be seen in Figure 17b; the background process of rule evaluation shown in Figure 17b and rule preview shown in Figure 16b is similar, the difference being that: during the rule evaluation process, after the computer device of the rule evaluation module in the image content understanding platform obtains the content understanding results of each sample image, it will also perform evaluation and statistical processing on the content understanding results of each sample image to obtain the evaluation results; the rule evaluation module will return the evaluation results to the front end of the platform for display.

[0175] It should be noted that this application embodiment also supports viewing rule snapshots of task rules that have undergone rule evaluation, so as to facilitate business users to view task rules that have undergone rule evaluation in batches, thereby improving the convenience and flexibility of task rule management. As shown in Figure 17c, the service interface provided by the image content logic platform includes an evaluation component 1702; when the evaluation component 1702 is selected, the service interface displays rule snapshots of task rules that have undergone rule evaluation in historical time. Here, a snapshot can be understood as a data backup technology. A rule snapshot is a data copy obtained by copying the state of a task rule at a certain moment; by viewing the rule snapshot, business users can quickly understand the state of the task rule at a certain moment, such as evaluation results, evaluation time, rule version identifier, and other information.

[0176] S804: Responds to content understanding tasks in business scenarios and obtains the task rules for content understanding tasks.

[0177] S805: Obtain the image feature vector of the image.

[0178] S806: Perform content understanding processing on the image feature vectors according to the task rules to obtain the content understanding results of the image.

[0179] It should be noted that the specific implementation process shown in steps S804-S806 is similar to the specific implementation process shown in steps S401-S403 in the embodiment shown in Figure 4 above. Please refer to the relevant description of the specific implementation process shown in steps S401-S403 above, which will not be repeated here.

[0180] In summary, the image content understanding platform in this embodiment possesses rich platform capabilities. Business users can leverage these capabilities to develop zero-sample prompts and fine-tune few-sample models, and configure personalized task rules for content understanding tasks. This significantly lowers the development threshold for image content understanding, thereby reducing the development costs required. In actual image content understanding processes, computer devices can directly perform content understanding processing on the image feature vectors according to the task rules, simplifying the complexity of image content understanding processing.

[0181] Figure 18 shows a schematic diagram of an image processing apparatus provided in an exemplary embodiment of this application; the image processing apparatus can be used to perform some or all of the steps in the method embodiments shown in Figures 3, 4, 7 and 8. Referring to Figure 18, the image processing apparatus includes: a content understanding module 1801, a rule module 1802 and a basis model reasoning module 1803;

[0182] The content understanding module 1801 is used to respond to content understanding tasks in business scenarios. It obtains task rules for the content understanding task from the rule module 1802. The content understanding task is used to instruct the image to be processed for content understanding. The task rules are used to instruct the processing rules to be used when performing content understanding processing. The task rules are pre-configured based on prompt words or fine-tuning models.

[0183] The basis model inference module 1803 is used to obtain the image feature vector of the image, which is used to characterize the image features of the image.

[0184] The content understanding module 1801 is also used to perform content understanding processing on the image feature vector according to the task rules to obtain the content understanding result of the image, which is used to indicate the content category to which the image belongs.

[0185] In one implementation, the task rules include at least one task condition and the relationships between task conditions. Each task condition defines a content category and a content understanding processing method corresponding to that content category. The relationships include parallel or combined usage relationships. The content understanding module 1801 is used to perform content understanding processing on the image feature vector according to the task rules. When obtaining the content understanding result of the image, it is specifically used for:

[0186] According to each task condition, the image feature vector is processed for content understanding to obtain the category result of the image under each task condition; the category result is used to represent: the image belongs to the content category defined by the task condition, or the image does not belong to the content category defined by the task condition.

[0187] Based on the relationships between task conditions, the category results of the image under each task condition are processed to generate the content understanding results of the image.

[0188] In one implementation, the task condition is a cue word condition. The content understanding processing method defined in the cue word condition includes: performing content understanding processing on the image using cue words and cue word thresholds; the content understanding module 1801 is used to perform content understanding processing on the image feature vector according to each task condition, and when obtaining the category result of the image under each task condition, it is specifically used for:

[0189] Obtain the word feature vector of the prompt word. The word feature vector is used to represent the semantic features of the prompt word.

[0190] The similarity between the image feature vector and the word feature vector is calculated to obtain the similarity result; the similarity result is used to characterize the degree of similarity between the image and the prompt word;

[0191] The similarity results are compared with the prompt word threshold to obtain the image category result under the prompt word condition;

[0192] Specifically, if the similarity result is greater than or equal to the prompt word threshold, the image category result under the prompt word condition is used to represent that the image belongs to the content category defined by the prompt word condition; if the similarity result is less than the prompt word threshold, the image category result under the prompt word condition is used to represent that the image does not belong to the content category defined by the prompt word condition.

[0193] In one implementation, the task conditions are fine-tuning model conditions. The content understanding processing methods defined in the fine-tuning model conditions include: performing content understanding processing on the image using the fine-tuning model and the model confidence of the fine-tuning model; the content understanding module 1801 is used to perform content understanding processing on the image feature vector according to each task condition, and when obtaining the category result of the image under each task condition, it is specifically used for:

[0194] The fine-tuning model is invoked to perform content understanding processing on the image feature vectors to obtain the image's probability information; the probability information is used to characterize the probability that the image belongs to the content category defined by the fine-tuning model.

[0195] The probability information of the image is compared with the model confidence to obtain the category result of the image under the fine-tuned model conditions;

[0196] Specifically, if the probability information is greater than or equal to the model confidence, the category result of the image under the fine-tuning model condition is used to characterize that the image belongs to the content category defined by the fine-tuning model condition; if the probability information is less than the model confidence, the category result of the image under the fine-tuning model condition is used to characterize that the image does not belong to the content category defined by the fine-tuning model condition.

[0197] In one implementation, the content understanding task is a review task, and the content understanding module 1801 is also used for:

[0198] Based on the content understanding results of the image, the image is processed for content review, and content review information is generated. The content review information is used to describe whether the image meets the requirements of the business scenario.

[0199] The business scenarios include at least one of the following: live video streaming, audio and video, and driving scenarios.

[0200] In one implementation, the business scenario corresponds to a sample library, and the content understanding module 1801 is also used for:

[0201] Acquire a data stream, which includes multiple target sample images;

[0202] Image analysis processing is performed on each target sample image to obtain sample data for each target sample image;

[0203] The sample data of each target sample image is stored in the sample library.

[0204] In one implementation, the sample library includes an index library and a storage library; the sample data includes sample identifiers and sample-related information, the sample-related information including sample feature vectors of sample images; the content understanding module 1801, used to store sample data of each sample image into the sample library, is specifically used for:

[0205] Store the sample identifier of each sample image and the library identifier of the sample library in the index library;

[0206] Each sample image, its sample identifier, and related information are stored in the repository.

[0207] In one implementation, the sample library includes a vector library, and the sample data includes sample feature vectors of sample images; the content understanding module 1801, when storing the sample data of each sample image into the sample library, is specifically used for:

[0208] Store the sample feature vector of each sample image into the vector library.

[0209] In one implementation, the content understanding module 1801 is also used for:

[0210] Obtain the prompt words and perform feature extraction processing on the prompt words to obtain the word feature vectors of the prompt words;

[0211] Calculate the matching information between word feature vectors and each sample image in the sample library; the sample library includes an index library and a storage library, or a vector library;

[0212] Based on the matching information between word feature vectors and sample images, a threshold for prompt words is set.

[0213] In one implementation, the content understanding module 1801 is also used for:

[0214] Multiple sample images were selected from the sample library;

[0215] If there is a first sample image without a label among the multiple sample images, then the label for the first sample image is received;

[0216] A training dataset is constructed based on the second sample image carrying the label and the first sample image with the label annotation from multiple sample images;

[0217] The classification model is fine-tuned based on the training dataset to obtain the fine-tuned model.

[0218] In one implementation, the task rule includes at least one task condition, and the content understanding module 1801 is further used for:

[0219] The display shows the rule configuration interface for the content understanding task, which includes the configuration area corresponding to the task rule;

[0220] Receive condition information for the task conditions set in the configuration area;

[0221] In response to the submission of conditional information, task rules are generated; and...

[0222] Generate a rule version identifier for the task rule; the rule version identifier is either the initial version number or generated based on the historical version number of the task rule.

[0223] In one implementation, the rule configuration interface includes a rule preview component; the content understanding module 1801 is also used for:

[0224] Receive the library identifier selected for the task rule in the rule configuration interface. The library identifier corresponds to a sample library.

[0225] In response to the trigger operation of the rule preview component, the content understanding processing of each sample image in the sample library is performed based on the task rules to obtain the content understanding results of each sample image;

[0226] Output the content understanding results for each sample image.

[0227] In one implementation, the rule configuration interface includes a rule evaluation component; the content understanding module 1801 is also used for:

[0228] Receive the library identifier selected for the task rule in the rule configuration interface. The library identifier corresponds to a sample library.

[0229] In response to the trigger operation of the rule evaluation component, the content understanding processing of each sample image in the sample library is performed based on the task rules to obtain the content understanding results of each sample image;

[0230] The content understanding results of each sample image are evaluated and statistically processed to obtain the evaluation results;

[0231] Output the evaluation results.

[0232] According to one embodiment of this application, the various units in the image processing apparatus shown in FIG18 can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by multiple units working together. According to another embodiment of this application, the image processing apparatus shown in FIG18 and the image processing method of the embodiments of this application can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding methods shown in FIG3, 4, 7 and 8 on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Computer programs can be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0233] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the image processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0234] Figure 19 shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. Referring to Figure 19, the computer device includes a processor 1901, a communication interface 1902, and a computer-readable storage medium 1903. The processor 1901, communication interface 1902, and computer-readable storage medium 1903 can be connected via a bus or other means. The communication interface 1902 is used to receive and send data. The computer-readable storage medium 1903 can be stored in the memory of the computer device and is used to store computer programs. The processor 1901 is used to execute the computer programs stored in the computer-readable storage medium 1903. The processor 1901 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more computer programs, specifically suitable for loading and executing one or more computer programs to achieve corresponding method flows or corresponding functions.

[0235] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, one or more computer programs suitable for loading and execution by the processor 1901 are also stored in this storage space. It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0236] In one embodiment, a computer-readable storage medium stores one or more computer programs; the processor 1901 loads and executes one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-described image processing method embodiment; specifically, the one or more computer programs in the computer-readable storage medium are loaded and executed by the processor 1901 to implement the knowledge question-answering method described above.

[0237] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the image processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0238] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described image processing method.

[0239] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0240] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. A computer program product includes one or more computer programs. When a computer program is loaded and executed on a computer device, the computer program performs the processes or functions described above in the embodiments of this application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in or transmitted through a computer-readable storage medium. The computer program can be transmitted from one website, computer device, server, or data center to another website, computer device, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible to the computer device or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0241] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method is performed by a computer device, and the method includes: In response to a content understanding task in a business scenario, the task rules for the content understanding task are obtained. The content understanding task is used to instruct content understanding processing of an image. The task rules are used to instruct the processing rules used when performing content understanding processing. The task rules are pre-configured based on prompt words or fine-tuning models. Obtain the image feature vector of the image, which is used to characterize the image features of the image; The image feature vector is processed for content understanding according to the task rules to obtain the content understanding result of the image, which is used to indicate the content category to which the image belongs.

2. The method as described in claim 1, characterized in that, The task rules include at least one task condition and the relationships between the task conditions. Each task condition defines a content category and a corresponding content understanding processing method. The relationships include parallel or combined usage relationships. The step of performing content understanding processing on the image feature vector according to the task rules to obtain the content understanding result of the image includes: According to each task condition, the image feature vector is processed by content understanding to obtain the category result of the image under each task condition; the category result is used to characterize whether the image belongs to the content category defined by the task condition or does not belong to the content category defined by the task condition. Based on the correlation between the task conditions, the category results of the image under each task condition are processed to generate the content understanding results of the image.

3. The method as described in claim 1 or 2, characterized in that, The task conditions are cue word conditions, and the content understanding processing method defined in the cue word conditions includes: performing content understanding processing on the image using cue words and cue word thresholds; and performing content understanding processing on the image feature vector according to each task condition to obtain the category result of the image under each task condition, including: Obtain the word feature vector of the prompt word, and the word feature vector is used to characterize the semantic features of the prompt word; The similarity between the image feature vector and the word feature vector is calculated to obtain a similarity result; the similarity result is used to characterize the degree of similarity between the image and the prompt word; The similarity result is compared with the prompt word threshold to obtain the category result of the image under the prompt word condition; Wherein, if the similarity result is greater than or equal to the prompt word threshold, the category result of the image under the prompt word condition is used to characterize that the image belongs to the content category defined by the prompt word condition; if the similarity result is less than the prompt word threshold, the category result of the image under the prompt word condition is used to characterize that the image does not belong to the content category defined by the prompt word condition.

4. The method according to any one of claims 1-3, characterized in that, The task conditions are fine-tuning model conditions. The content understanding processing method defined in the fine-tuning model conditions includes: performing content understanding processing on the image using the fine-tuning model and the model confidence of the fine-tuning model; and performing content understanding processing on the image feature vector according to each task condition to obtain the category result of the image under each task condition, including: The fine-tuning model is invoked to perform content understanding processing on the image feature vector to obtain the probability information of the image; the probability information is used to characterize the probability that the image belongs to the content category defined by the condition of the fine-tuning model. The probability information of the image is compared with the confidence level of the model to obtain the category result of the image under the fine-tuned model conditions; Wherein, if the probability information is greater than or equal to the model confidence, the category result of the image under the fine-tuning model condition is used to characterize that the image belongs to the content category defined by the fine-tuning model condition; if the probability information is less than the model confidence, the category result of the image under the fine-tuning model condition is used to characterize that the image does not belong to the content category defined by the fine-tuning model condition.

5. The method according to any one of claims 1-4, characterized in that, The content understanding task is a review task, and the method further includes: Based on the content understanding results of the image, the image is subjected to content review processing to generate content review information for the image; the content review information is used to describe whether the image meets the scenario requirements of the business scenario; The business scenarios include at least one of the following: live video streaming, audio and video, and driving scenarios.

6. The method according to any one of claims 1-5, characterized in that, The business scenario corresponds to a sample library, and before responding to the content understanding task in the business scenario, it also includes: Acquire a data stream, which includes multiple sample images; Image analysis processing is performed on each of the sample images to obtain sample data for each sample image; The sample data of each sample image is stored in the sample library.

7. The method according to any one of claims 1-6, characterized in that, The sample library includes an index library and a storage library; the sample data includes sample identifiers and sample-related information, the sample-related information including sample feature vectors of the sample images; storing the sample data of each sample image into the sample library includes: The sample identifier of each sample image and the library identifier of the sample library are stored in the index library; Each sample image, its sample identifier, and related sample information are stored in the repository.

8. The method according to any one of claims 1-7, characterized in that, The sample library includes a vector library, and the sample data includes sample feature vectors of the sample images; storing the sample data of each sample image into the sample library includes: The sample feature vector of each sample image is stored in the vector library.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Obtain the prompt words and perform feature extraction processing on the prompt words to obtain the word feature vector of the prompt words; The matching information between the word feature vector and each of the sample images in the sample library is calculated respectively; the sample library includes an index library and a storage library, or a vector library; Based on the matching information between the word feature vector and each of the sample images, a prompt word threshold is set for the prompt word.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Multiple target sample images are selected from the sample library; If there is a first sample image without a label among the multiple target sample images, then the label for the first sample image is received; A training dataset is constructed based on the second sample image carrying the label and the first sample image with the label in the multiple target sample images; The classification model is fine-tuned based on the training dataset to obtain the fine-tuned model.

11. The method according to any one of claims 1-10, characterized in that, The task rule includes at least one task condition, and prior to the response to the content understanding task in the business scenario, it also includes: The rule configuration interface for the content understanding task is displayed, and the rule configuration interface includes the configuration area corresponding to the task rule; Receive condition information for the task conditions set in the configuration area; In response to the submission of the condition information, the task rule is generated; and... Generate a rule version identifier for the task rule; the rule version identifier is either an initial version number or generated based on a historical version number of the task rule.

12. The method according to any one of claims 1-11, characterized in that, The rule configuration interface includes a rule preview component; after receiving the condition information of the task conditions set in the configuration area, it also includes: Receive the library identifier selected for the task rule in the rule configuration interface, where the library identifier corresponds to a sample library; In response to a trigger operation on the rule preview component, content understanding processing is performed on each sample image in the sample library based on the task rule to obtain the content understanding result of each sample image; Output the content understanding results for each of the sample images.

13. The method according to any one of claims 1-12, characterized in that, The rule configuration interface includes a rule evaluation component; after receiving the condition information of the task conditions set in the configuration area, it also includes: Receive the library identifier selected for the task rule in the rule configuration interface, where the library identifier corresponds to a sample library; In response to a trigger operation on the rule evaluation component, content understanding processing is performed on each sample image in the sample library based on the task rules to obtain the content understanding results of each sample image; The content understanding results of each sample image are evaluated and statistically processed to obtain the evaluation results; Output the evaluation results.

14. An image processing apparatus, characterized in that, The image processing device is mounted on a computer device and includes: a content understanding module, a rule module, and a basis model inference module; The content understanding module is used to respond to content understanding tasks in business scenarios, obtain task rules for the content understanding task from the rule module, the content understanding task is used to instruct content understanding processing of the image; the task rules are used to instruct the processing rules used when performing content understanding processing, and the task rules are pre-configured based on prompt words or fine-tuning models; The basis model inference module is used to obtain the image feature vector of the image, and the image feature vector is used to characterize the image features of the image. The content understanding module is further configured to perform content understanding processing on the image feature vector according to the task rules to obtain the content understanding result of the image, and the content understanding result is used to indicate the content category to which the image belongs.

15. The apparatus as claimed in claim 14, characterized in that, The image processing device also includes a hosting module, which deploys a fine-tuned model after model fine-tuning processing; The managed module is used to call the fine-tuning model to perform content understanding processing on the image feature vector to obtain the probability information of the image; The probability information is used to characterize the probability that the image belongs to the content category defined by the fine-tuning model conditions; The content understanding module is also used to compare the probability information of the image with the model confidence level to obtain the category result of the image under the fine-tuning model conditions; Wherein, if the probability information is greater than or equal to the model confidence, the category result of the image under the fine-tuning model condition is used to characterize that the image belongs to the content category defined by the fine-tuning model condition; If the probability information is less than the model confidence level, then the category result of the image under the fine-tuning model condition is used to characterize that the image does not belong to the content category defined by the fine-tuning model condition.

16. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the image processing method as described in any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-13.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1-13.

Images