Rapid interventions for deployed machine learning systems using few-shot signatures

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for modulating an output of a deployed machine learning system. In one aspect a method comprises maintaining a set of one or more signatures that each represent a respective set of inputs to a base machine learning model that have been determined to degrade a performance of the base machine learning model, receiving a new input to the base machine learning model, for each of the set of one or more signatures, processing the new input and the signature using an auxiliary machine learning model to generate a respective auxiliary output that indicates whether the new input belongs to the respective set of inputs represented by the signature, and modulating an output of the base machine learning model for the new input based on the respective auxiliary outputs for the set of one or more signatures.

Description

[0001] Attorney Docket No. 56113-0839WO1

[0002] RAPID INTERVENTIONS FOR DEPLOYED MACHINE LEARNING SYSTEMS USING FEW-SHOT SIGNATURES

[0003] BACKGROUND

[0004] [1] This specification relates to processing data using machine learning models.

[0005] [2] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0006] [3] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

[0007] SUMMARY

[0008] [4] This specification describes a deployed machine learning system implemented as computer programs on one or more computers in one or more locations that can modulate the output of a base machine learning model using a set of one or more signatures. In particular, the system can modulate the output of the base machine learning model, e.g., a base computer vision machine learning model, by using an auxiliary machine learning model to determine whether a new input, e.g., an input image, includes content represented by the set of signatures.

[0009] [5] In this specification, each signature in the set of signatures represents a respective set of inputs to the base machine learning model that degrades the performance of the base machine learning model. For example, a signature can represent a novel input which was not seen by the base machine learning model during training but has been identified as an unsupported input to the deployed machine learning system that resulted in degraded, e.g.. poor or unexpected, model performance. More specifically, an entity, e.g., the system, another system, an administrator, a user, etc., can identify one or more inputs to the deployed machine learning system as candidates for creating a signature based on the tendency of the one or more inputs to confound the base machine learning model.

[0010] [6] According to a first aspect there is provided a method for maintaining a set of one or more signatures that each represent a respective set of inputs to a base machine learning model that have been determined to degrade a performance of the base machine learning model, receiving a new input to the base machine learning model, for each of the set of one or moreAttorney Docket No. 56113-0839WO1

[0011] signatures, processing the new input and the signature using an auxiliary' machine learning model to generate a respective auxiliary output that indicates whether the new input belongs to the respective set of inputs represented by the signature, and modulating an output of the base machine learning model for the new input based on the respective auxiliary' outputs for the set of one or more signatures.

[0012] [7] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0013] [8] The system of this specification can provide for the enhanced performance of a deployed machine learning system by modulating an output of the deployed machine learning system, without requiring retraining or finetuning of the deployed machine learning system. In particular, the system is able to capture the salient traits of inputs that have been shown to degrade the performance of a base machine learning model using a set of signatures that can be rapidly deployed to modulate the output of the base machine learning model. The deployed machine learning system can therefore significantly reduce the time required to address newly discovered vulnerabilities and improve its ability to handle a wider range of inputs, including those that may have initially caused performance degradation in the base machine learning model since they were not seen during training, by leveraging the set of signatures.

[0014] [9] In particular, instead of retraining the base machine learning model, the system can process a new input and each of the signatures in the set of signatures using an auxiliary machine learning model to generate respective auxiliary’ outputs that indicate whether the new input includes content represented by the signature. Thus, the system can enhance the performance of the deployed base machine learning model by modulating the output based on the signatures, e g., in order to detect and respond appropriately to malicious, undesired, or unsupported inputs.

[0015]

[0010] More specifically, the system of this specification can reduce the use of computational resources needed to provide performance gains relative to a system that is required to retrain the base machine learning model when one or more new sets of inputs that degrade the performance of the base machine learning model are identified. Since a machine learning model, e.g., a computer vision model, can have many parameters, e g., thousands, hundreds of thousands, or millions of parameters, the amount of computational resources, including processing power and memory, required to update the parameter values can be significant. Furthermore, in the case that the base machine learning model is deploy ed on a user device, the computational resources required to frequently push the updated model parameters to theAttorney Docket No. 56113-0839WO1

[0016] user device can be nontrivial. Thus, it is generally costly, and sometimes infeasible, to retrain a deployed machine learning model when one or more new sets of inputs that degrade the performance are identified. By providing an alternate method for modifying the performance of the system that does not require training or finetuning, the system can close the performance gap on inputs the system is potentially vulnerable to using the set of signatures, e.g., by modulating the outputs of the base machine learning model, while decreasing the computational resources necessary to achieve the performance gains.

[0017]

[0011] Moreover, in some cases, the auxiliary machine learning model can be implemented as a lightweight machine learning model, e.g., a machine learning model that has less parameters, a less complicated architecture, or both, than the base machine learning model. In this case, the computational resources, e.g., processing cycles or time, memory bandwidth, network bandwidth, and so on, required to train the auxiliary machine learning model to determine whether a new input includes content represented by a signature are fewer than the computational resources required to train or finetune the base machine learning model.

[0018]

[0012] Additionally, the techniques of this specification enable the generation of signatures using a generative machine learning model that is configured to process one or more of a text description, e.g., a natural language description, of an input that has been determined to degrade the base machine learning model performance or a set of one or more example images characterizing the input to generate a signature embedding that can be used as the signature. Thus, the techniques facilitate obtaining more data for use in signature generation, e.g.. by augmenting a small or non-existent set of available examples of inputs that have been determined to degrade the base machine learning model. In particular, the system can require fewer data examples to adapt to new scenarios compared to traditional model retraining approaches, e.g., the system, or another system, can generate signatures using natural language descriptions, a minimal number of inexpensive image examples, or both. Since generating the signatures requires at least a text description or a set of a few example images, the system can avoid data collection set-up costs and also save valuable human time and effort by not requiring extensive data collection of a large dataset representative of each new set of inputs that degrade the performance of the base machine learning model to modify the base machine learning model’s performance. In some cases, the system, or another system, can use a vision-language model to generate signatures in a joint text-image embedding space, thereby capturing salient traits of malicious, undesired, or unsupported inputs more effectively than traditional approaches. In particular, by jointly leveraging the text description and the example images to generate a signature embedding, the techniques of this specification can provide for theAttorney Docket No. 56113-0839WO1

[0019] generation of a more robust signature embedding with respect to a signature generated with either the text description or the set of example images alone. In some cases, using signature embeddings generated by a vision-language model can lead to more robust modulation of a wider range of input scenarios without requiring frequent base machine learning model updates.

[0020]

[0013] In addition, the system can enable efficient deployment of the set of signatures for interventions that address new vulnerabilities and close performance gaps. In the case that the system is implemented on a user device, the system can obtain signatures via over-the-air updates from a server. This capability can allow7for rapid distribution of new7signatures to the deployed system without requiring updates to the base machine learning model. The rapid intervention capabilities provide enhanced flexibility in responding to emerging issues in unsupported inputs that are identified in inputs to the system.

[0021]

[0014] Furthermore, the system of this specification can improve the security of the overall system on which the base machine learning model is deployed, since any vulnerabilities in the model can negatively impact downstream components of the overall system. In particular, one or more additional machine learning models or systems can receive inputs from or can be activated based on the output of the deployed machine learning system. The system of this specification can allow7for the signatures to be deployed as a rapid intervention in order to maintain security while more comprehensive solutions are being developed. Updating the base machine learning model can be a time-intensive process, e.g., that includes collecting data for retraining. In contrast, deploying the signatures can help preserve overall system security by protecting the dow nstream components in the face of evolving challenges that pose a threat to the overall system, without relying on or waiting for retraining or updating the base machine learning model to modify the system's response to the unsupported inputs.

[0022]

[0015] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below7. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0023] BRIEF DESCRIPTION OF FIGURES

[0024]

[0016] FIG. 1 is a system diagram of an example deployed computer vision system that includes a modulation engine.Atorney Docket No. 56113-0839WO1

[0025]

[0017] FIGS. 2A-2B illustrate an example implementation of a deployed computer vision system in which a user device receives a set of signatures or resolution data from a server, thereby changing the state of the modulation engine.

[0026]

[0018] FIG. 3 demonstrates an example of generating a signature using a text description and a set of example images.

[0027]

[0019] FIG. 4 is a flow diagram of an example process for generating a modulated output in a deployed machine learning system based on a set of signatures.

[0028]

[0020] Like reference numbers and designations in the various drawings indicate like elements.

[0029] DETAILED DESCRIPTION

[0030]

[0021] FIG. 1 shows an example deployed computer vision system 100. The deployed computer vision system 100 is an example of a deployed machine learning system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0031]

[0022] Typically, a computer vision system includes a computer vision model that operates by extracting features from input data and using those features to make predictions or classifications. However, such models may have limitations in handling novel or unsupported inputs, e.g., that were not well-represented in the training data set used to train the computer vision model. For example, a computer vision model trained to detect vehicles may struggle with images of new vehicle designs or vehicles in unusual lighting conditions that are out of the distribution of data seen in the training data set. As another example, a computer vision model trained to verify that photo uploads to a photo sharing site comply with the guidelines of the site can be confounded by images that include a new symbol or sign that was out of the distribution of the data seen in the training data set.

[0032]

[0023] To address these limitations, the deployed computer vision system 100 utilizes a set of signature(s) 132 to represent novel inputs that have been determined to degrade the performance of the computer vision model. In particular, the deployed computer vision system 100 depicted in FIG. 1 can receive and process a new input 105 to modulate the output of the base machine learning model 120 based on the set of one or more signature(s) 132. As an example, the new input 105 can include an image, e.g., a digital photograph, or a video, e.g., a video stream, or other visual data. The modulated output 145 can account for malicious, undesired, or unsupported content in the new input 105. as is described in more detail below.

[0033]

[0024] The system 100 can include a base machine learning model 120 that is configured to process an input, e g., the new input 105, to generate a base output 140 in accordance with theAttorney Docket No. 56113-0839WO1

[0034] computer vision task that the base machine learning model 120 has been trained for. In some cases, the base machine learning model 120 can be a computer vision model optimized for specific tasks.

[0035]

[0025] For example, the base output 140 can include a numerical value, text in a natural or computer language, or both that defines a result of the task, e.g., for tasks such as image captioning, object detection, optical character recognition, or segmentation. In some cases, the base output 140 can include a binary indicator that defines whether or not the task was successfully completed. As another example, the base output 140 can include a vector of values that define the next configuration of a robot in a pose estimation task, or defines a classification of an identified action in an action detection task. As yet another example, the base model output 140 can define an intermediate step to be performed during the task, e.g.. a call to a software API for a software tool that is used to complete the task.

[0036]

[0026] The base machine learning model 120 can have any appropriate machine learning architecture that can be configured to process the new input 105 to generate the base output 140 in accordance with the computer vision task that the base machine learning model 120 was trained for. In particular, the base machine learning model 120 can directly generate the base output 140 without modulation, representing the unmodulated processing result of the system 100 for a given new input 105.

[0037]

[0027] For example, the base machine learning model 120 can be a random forest, a support vector machine, a decision tree, or a neural network. In the case that the base machine learning model 120 is a neural netw ork, the base machine learning model can be a neural network with any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully -connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0038]

[0028] For example, the base machine learning model 120 can be a convolutional neural network (CNN), e.g., one having an AlexNet or Residual Network (ResNet) architecture that has been trained for an object detection task or an image classification task. As another example, the base machine learning model 120 can be a You Only Look Once (YOLO) model that has been trained for a segmentation task. As yet another example, the base machine learning model 120 can be an ALIGN (A Large-scale ImaGe and Noisy-text embedding) model, e.g., as is described in Jia, C., et. al. “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision’" (arXiv:2102.05918v2), that has been trained for an image-text retrieval task.Attorney Docket No. 56113-0839WO1

[0039]

[0029] As a further example, the base machine learning model 120 can be implemented as a vision transformer (ViT) model, e.g.. an encoder-decoder transformer, an encoder-only transformer, or a decoder-only transformer, configured to perform parallel processing of the contents of the multimodal input using a multi-headed attention mechanism. For example, the base machine learning model 120 can be implemented as an End-to-End Detection Transformer (DETR) for an obj ect detection task.

[0040]

[0030] The system 100 can evaluate whether the base output 140 of the base machine learning model 120 should be modulated using a modulation engine 130. In the particular example depicted, the modulation engine 130 is activated, e.g., as is represented by the grayed box. When activated, the modulation engine 130 can process new inputs received by the system 100 to modulate the output of the system 100 based on the set of signature(s) 132. In particular, the system 100 can provide two types of outputs: a base output 140 and a modulated output 145, in response to a new input 105. The base output 140 can be generated directly from the base machine learning model 120, while the modulated output 145 can be generated by the activated modulation engine 130 based on the set of signature(s) 132.

[0041]

[0031] More specifically, the modulation engine 130 can include a set of signature(s) 132 and an auxiliary machine learning model 134. In this specification, each signature in the set of signature(s) 132 represents a respective set of inputs to the base machine learning model 120 that have been determined to degrade the performance of the base machine learning model 120. For example, a signature can be a data structure, e.g., a vector, a tensor, an embedding, or a latent space representation, etc., that represents a novel type of input. In some cases, the novel type of input was not seen by the base machine learning model 120 during training but has been identified as an unsupported input to the deployed computer vision system 100, e.g., since the novel input results in poor or unexpected performance of the base machine learning model 120.

[0042]

[0032] In particular, the modulation engine 130 can maintain a current set of signature(s) 132 that the engine 130 uses to determine whether modulation is necessary' for a particular input, e.g., the new input 105. For example, the modulation engine 130 can obtain the set of signature(s) 132 by generating the set of signature(s) 132 or by receiving the set of signature(s) 132 from another system. Furthermore, in some cases, the set of signature(s) 132 can be updated, e.g., by obtaining one or more new signatures to add to the current set of signature(s) 132 maintained by the modulation engine 130.

[0043]

[0033] As an example, the system 100, or another system, can identify the sets of input(s) 154 for use in generating the signature(s) 132. As another example, an administrator or a user ofAttorney Docket No. 56113-0839WO1

[0044] the deployed computer vision system 100 can identify the set of inputs for use in generating the signature(s) 132. The sets of input(s) 154 can be identified at any appropriate time, e.g.. the set of input(s) 154 can be identified independently from the deployed computer vision system 100 receiving inputs, e.g., the new input 110. In particular, the sets of input(s) 154 represented by the signatures can include inputs that have resulted in degraded base machine learning model 120 performance.

[0045]

[0034] In the particular example depicted, the deployed computer vision system 100 receives the set of signature(s) 132 from a signature generation engine 150. In some cases, the signature generation engine 150 is part of the deployed computer vision system 100. In other cases, the signature generation engine 150 is part of a different system. An example in which the signature generation engine 150 is implemented on a server and the deployed computer vision system 100 is implemented on a user device will be described in more detail with respect to FIGS. 2A-2B.

[0046]

[0035] For example, the signature generation engine can include a generative machine learning model 152 that can be used to process each set of input(s) in the sets of input(s) 154 to generate each signature in the corresponding set of signature(s) 132. In particular, the generative machine learning model 152 can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully -connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0047]

[0036] In some cases, each signature in the set of signature(s) 132 is a signature embedding, e.g., a collection of values that represents a particular set of inputs that have been determined to degrade the performance of the base machine learning model 120 in an embedding space.

[0048]

[0037] As an example, the signature generation engine 150 can process each input in a particular set of inputs using the generative machine learning model 154 to obtain an intermediate output of the generative machine learning model 154 as a respective input embedding and can aggregate the respective input embeddings as the signature embeddings. In some cases, a signature embedding can be a single vector that represents the corresponding set of input(s). e.g.. by averaging or clustering the respective input embeddings for the set of input(s) as the signature embedding. In other cases, a signature embedding can be a matrix of vectors, e.g., by concatenating the respective input embeddings for the set of input(s) as the signature embedding.

[0049]

[0038] As another example, the signature generation engine 150 can generate the signature embeddings without generating and aggregating intermediate input embeddings. For example,Attorney Docket No. 56113-0839WO1

[0050] the set of signature(s) 132 can be generated by generating embeddings using the generative machine learning model 154. In this case, the generative machine learning model can have any appropriate neural network architecture that can be configured to process an input that characterizes the set of inputs that have been demonstrated to result in unexpected or poor base machine learning model 120 performance, e.g., a text description of the set of inputs in a zeroshot approach, a set of example inputs in a few-shot approach, or both, to generate a signature embedding.

[0051]

[0039] As an example, the generative machine learning model 152 can be a generative-adversarial network or an autoregressive language processing network. In particular, the generative machine learning model can have a recurrent neural network architecture that is configured to sequentially process contents of an input, and trained to perform next element prediction, e.g., to define a likelihood score distribution over a set of next elements. More specifically, the generative machine learning model can be a recurrent neural network (RNN), long short-term memory (LSTM), or gated-recurrent unit (GRU). As a further example, the generative machine learning model 152 can be transformer-based, e.g.. an encoder-decoder transformer, an encoder-only transformer, or a decoder-only transformer, configured to perform parallel processing of the contents of the input using a multi-headed attention mechanism. In some cases, the generative machine learning model 152 can be a visionlanguage model (VLM) that is configured to generate signature embeddings in a joint textimage space, as is described in more detail with respect to FIG. 3.

[0052]

[0040] In some cases, the modulation engine 130 can be inactive, e.g., in a default state, and is only activated upon receiving the set of signature(s) 132. As an example, the modulation engine 130 can be activated in response to receiving a set of signature(s) 130 in an over-the-air update from a server, e.g., by receiving a data update via a wireless connection as is described in more detail with respect to FIG. 2A. Furthermore, the modulation engine 130 can be deactivated upon determining that the set of signature(s) 130 is empty or in response to receiving resolution data from the server that indicates that the base machine learning model has been updated to achieve better performance on the sets of inputs represented by the signatures 130, e.g.. as will be described in more detail with respect to FIG. 2B.

[0053]

[0041] For example, when the system 100 receives a new input 105, the system 100 can process the new input 105 using the base machine learning model 120 and the auxiliary7machine learning model 134. More specifically, the modulation engine 130 can process the new input 105 and the set of signature(s) 132 using the auxiliary machine learning model 134 to generate respective auxiliary output(s) 136 that each indicate whether the new input 105 belongs to aAttorney Docket No. 56113-0839WO1

[0054] respective set of inputs represented by a particular signature. The deployed computer vision system 100 can then provide either a base output 140 or a modulated output 145 for a given input e.g., the new input 105, based on the respective auxiliary output(s) 136 generated for each of the signature(s) 132.

[0055]

[0042] In some cases, the system 100 can process the new input 105 concurrently using both the base machine learning model 120 and the auxiliary machine learning model 134. In another case, the system 100 can first process the new input 105 using the auxiliary machine learning model 134 to determine whether or not modulation is necessary based on the signature(s) 132. As an example, in the case that the engine 130 uses the auxiliary output(s) 136 to override the base output 140, e.g., as is described in more detail below, the system 100 can bypass the need to process the new input 105 using the base machine learning model 120 by generating the modulated output 145 directly from the auxiliary output(s) 136.

[0056]

[0043] In particular, the modulation engine 130 can process the new input 105 and the set of signature(s) 132 to determine whether modulation is necessary for the new input 105. For example, each auxiliary output in the output(s) 136 can be a binary indicator that indicates whether the new input 105 is represented by a particular signature. As another example, each auxiliary output in the output(s) 136 can be a tuple including the binary indicator and the same type of output as the base output 140, e.g., so the engine 130 can override the base output 140 using the auxiliary output for a particular signature in the case that the new input 105 belongs to the particular signature. As yet another example, each auxiliary output in the output(s) 136 can be a conditioning input, e.g., a class label or text describing why the base output is degraded based on the particular signature that the new input 105 belongs to, that can be further processed by the base machine learning model 140 to generate the modulated output 145, as will be described in more detail below.

[0057]

[0044] The auxiliary machine learning model 134 can be a neural network with any appropriate machine learning architecture that can be configured to process the new input 105 and one or more signatures from the set of signature(s) 132 to generate the respective auxiliary output(s) 136. For instance, the auxiliary machine learning model 134 can be a neural network with any appropriate number of neural network layers (e.g., 1 layer. 5 layers, or 10 layers) of any appropriate type (e.g., fully -connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers). In some cases, the auxiliary machine learning model 134 can be implemented as a lightweight machine learning model, e.g.. a machine learning model that has less parameters, a less complicated architecture, or both, than the base machine learning model 120.Attorney Docket No. 56113-0839WO1

[0058]

[0045] The auxiliary machine learning model 134 can determine whether the new input 105 is characterized by any of the signature(s) 132, e.g., by comparing a representation of the new input 105 to each signature in the signature(s) 132. For example, in the case that the signature(s) 132 are signature embeddings, the auxiliary machine learning model 134 can process the new input 105 to generate an embedding of the new input 105 in an embedding space, and can determine a measure of discrepancy between the embedding of the new input and each signature embedding. The auxiliary machine learning model 134 can then evaluate whether the discrepancy between the embedding of the new input 105 and a particular signature embedding satisfies a threshold discrepancy criterion. For example, the measure of discrepancy can be a measure of similarity.

[0059]

[0046] In some cases, the threshold discrepancy criterion can be a tunable parameter, e.g., that can be tuned without requiring retraining or fine-tuning of the base machine learning model 120. In particular, a validation system can tune the tunability parameter using a validation data set, e.g., to allow for adjustment of the sensitivity' of the modulation process with respect to the validation set.

[0060]

[0047] In particular, the measure of discrepancy can be any appropriate distance measure between the embedding of the new input 105 and the signature embedding in the embedding space, and the threshold discrepancy criterion can be a threshold distance criterion. As an example, the auxiliary machine learning model 134 can compute a Euclidean distance, a cosine similarity distance, a Mahalanobis distance, or a Hamming distance between the embedding of the new input and each signature embedding. In this case, the auxiliary’ machine learning model 134 can evaluate whether the distance between the embedding of the new input 105 and a particular signature embedding satisfies a threshold distance criterion to determine whether the new input 105 includes content represented by the particular signature.

[0061]

[0048] In the case that the signature embeddings were generated using a vision-language model, the auxiliary machine learning model 134 can also be implemented as avision-language model. In this case, the deployed computer vision system 100 can process both the new input and the signature using the vision-language model, e.g., with an instruction to identify whether the new input 105 has content characterized by the signature. In the case that the visionlanguage model is a text and image embedding neural network that has been trained through a representation learning objective, e.g., contrastive learning, the system 100 can embed the new input 105 and compare it to the signature embedding without an instruction. As an example, the vision-language model can generate a similarity score as the measure of discrepancy for each pair of a new input and a particular signature in the set of signature(s). The model 134 canAttorney Docket No. 56113-0839WO1

[0062] then compare the similarity' score to the threshold discrepancy criterion to determine whether the new input 105 includes content represented by the particular signature.

[0063]

[0049] In particular, the auxiliary machine learning model 134 can generate an embedding of the new input 105 in the same joint text-image space as the signature embeddings, e.g., in order to facilitate the comparison of the input embedding to the signature embeddings. For example, the auxiliary machine learning model 134 can be the same vision-language model used to generate the signature embeddings, e.g.. as is described in more detail with respect to FIG. 3.

[0064]

[0050] In the case that the discrepancy satisfies the threshold discrepancy criterion for at least one signature in the set of signature(s) 132, the auxiliary' machine learning model 134 can generate a respective auxiliary output 136 that indicates that the new input 105 includes content that is represented by the particular signature, e.g., that the new input 105 belongs to the set of inputs that are represented by the particular signature. In some cases, the auxiliary output can additionally indicate a confidence level associated with the determination of whether the new input 105 has content represented by the signature, e.g., the confidence level can be or can be derived from the measure of discrepancy for the signature. In the case that the discrepancy does not satisfy the threshold discrepancy criterion for the particular signature in the set of signatures 132, the respective auxiliary^ output 136 can indicate that the new input 105 does not include content that is represented by the particular signature.

[0065]

[0051] In response to determining that the new input 105 includes content represented by at least one signature in the set of signature(s) 132 based on the auxiliary output(s) 136, the modulation engine 130 can modulate the base output 140 to generate a modulated output 145. In particular, the activated modulation engine 130 can use the auxiliary output(s) 136 to modulate the output of the base machine learning model 120 for the new input 105 using any appropriate modulation technique, e.g., depending on the specific implementation and requirements of the deployed computer vision system 100.

[0066]

[0052] For example, the activated modulation engine 130 can override the base output 140 of the base machine learning model 120 using the auxiliary output 136 of the auxiliary machine learning model 134. In some cases, the auxiliary’ output 136 can include an output of the same type as the base output 140. e.g., natural language text, a binary indicator, a vector of values, etc., that can be provided as the modulated output 145. In other cases, the auxiliary output 136 can be an indication that the base machine learning model 120 cannot reliably generate an output for the given new input 105, e.g., the system 100 can provide an error message to a user of the system 100 as the modulated output 145.Atorney Docket No. 56113-0839WO1

[0067]

[0053] As an example, the engine 130 can override the base output 140 in response to the auxiliary model 134 detecting a potential threat in the base output 140. As an example, potential threats can be identified based on their inclusion on a predetermined list, e.g., that is received from a user or administrator, and each identified threat can be represented by t a respective signature. In the case that the engine 130 determines that the base output 140 includes content that matches a potential threat based on a signature, the system 100 can discard the base output 140 and provide the auxiliary output for the particular signature that the modulation engine 130 determined that the new input 105 belonged to as the modulated output 145, e g., to a user.

[0068]

[0054] As another example, the activated modulation engine 130 can combine the base output 140 of the base machine learning model 120 with the corresponding auxiliary output 136 from the auxiliary machine learning model 134 for the signature. In this case, the auxiliary output 136 includes an output of the same type as the base output 140, e.g., the outputs 140 and 136 can be numerical values and combining the outputs can involve aggregating, e.g., averaging the outputs 140 and 136 from both models 120 and 134. In some cases, the aggregation can additionally involve a weighted combination, e.g.. based on confidence scores or other factors.

[0069]

[0055] As yet another example, while not depicted in FIG. 1, the activated modulation engine 130 can generate anew output using the base machine learning model 120, e.g., based on the corresponding auxiliary output 136 for the particular signature that the modulation engine 130 determined that the new input 105 belonged to. In particular, the engine 130 can condition the base machine learning model 120 by providing the auxiliary output 136 to the base machine learning model 120 with an instruction to generate the output based on the auxiliary output included as a conditioning input.

[0070]

[0056] As a further example, the activated modulation engine 130 can implement different fusion techniques between the outputs 140 and 136. For example, the auxiliary model 134 can generate a binary directive, e.g., an indication of approval or disapproval to influence the output of the base machine learning model 120, as the auxiliary output 136. In this case, the system can provide the base output 140 if the auxiliary output 136 indicates approval. As another example, the engine 130 can fuse the outputs 140 and 136 through more complex integration methods. In particular, in the case that the auxiliary machine learning model 134 is configured to additionally generate a confidence level associated with the determination of whether the new input 105 includes content represented by each signature, the engine 130 can fuse the outputs 140 and 136 based on the confidence level included in the auxiliary' output 136.

[0071]

[0057] In the case that the auxiliary machine learning model 134 determines that the discrepancy does not satisfy the threshold discrepancy criterion for any of the signature(s), theAttorney Docket No. 56113-0839WO1

[0072] respective auxiliary' output(s) 136 for each of the set of signature(s) 132 can indicate that the new input 105 does not include content that is represented by any of the signature(s) 132. In this case, the system 100 can provide the base output 140 as the final output, e.g., to a user. This allows the system 100 to maintain standard behavior for inputs that do not trigger any intervention through modulation.

[0073]

[0058] More specifically, by using the auxiliary’ machine learning model 134 to identify potentially unsupported inputs, the system 100 can modify the base output f40 without modifying the base machine learning model 120. Thus, the deployed computer vision system 100 can address performance degradation for new inputs represented by the set of signature(s) 132 without requiring training or fine-tuning of the base machine learning model 120, thereby allowing for rapid adaptation to new types of unsupported inputs, while maintaining the overall performance characteristics of the base machine learning model 120 through modulation.

[0074]

[0059] Furthermore, the modulation process can also include generating an alert to report a potential attack or violation, e.g., to another system or an administrator. For example, the deployed computer vision system 100 can provide an alert to a moderator for further review of the input. In this case, the alert can include information about which signature the new input 105 matched and, in some cases, the confidence level of the match. As another example, if the system 100 detects that a user is continually attempting to upload content that matches signatures associated with malicious or prohibited inputs, e.g., hate speech, the system 100 can restrict access by blocking the user from the system.

[0075]

[0060] While not depicted here, in some cases, instead of using a separate auxiliary machine learning model 134, the system 100 can use the signature(s) 132 to finetune the base machine learning model 120. In particular, the system, or another finetuning system, can update the base model's parameters based on the signature information, allowing it to better handle the problematic input types represented by the signatures. In this case, finetuning may be performed periodically or in response to the detection of a set of new input types that degrade the performance of the base machine learning model 120.

[0076]

[0061] The deployed computer vision system 100 can be applied in various real-world scenarios, such as for content moderation, user authentication, or for detecting new objects in a deployed autonomous vehicle system. In particular, by modulating the output using a modulation engine 130, the deployed computer vision system 100 can effectively adapt its behavior to handle novel or unsupported inputs represented by the set of signatures 132, thereby enhancing system 100 performance and robustness for a wide range of inputs.Atorney Docket No. 56113-0839WO1

[0077]

[0062] For example, the deployed computer vision system 100 can be deployed as a user authentication system. In some cases, the deployed computer vision system 100 can be configured to detect unwanted symbols or objects in the background of images including faces. For instance, the base machine learning model 120 can be trained to perform user authentication, e.g., to unlock a user device, while the set of signatures 132 can include representations of effects that have been found to confound the base machine learning model 120 and result in degraded performance, e.g.. cluttered backgrounds, objects with human features, e.g., mannequins, bright lights, reflective background surfaces, etc. In this case, the system can process both the face detection results as the base output 140 and each signature in the set of signature(s) 134 using the modulation engine 130 to determine whether any problematic elements are present in the image background and can correct for the identified problematic elements by generating a modulated output 145. In some cases, the modulated output is an error that alerts the user to which problematic elements are present.

[0078]

[0063] As another example, the deployed computer vision system 100 can be deployed for content moderation, e.g., on a photo-sharing application. In this case, the base machine learning model 120 can be trained to classify images as appropriate or inappropriate for general viewing on the platform or photo-sharing application, and the set of signature(s) 132 can include representations of newly identified types of content that have been found to degrade the performance of the base machine learning model 120. For example, the set of signature(s) can include potentially offensive or dangerous symbols. When a user attempts to upload an image, the deployed computer vision system 100 can process the image using both the base machine learning model 120 and the modulation engine 130 to determine whether the content should be flagged or blocked.

[0079]

[0064] As yet another example, the deployed computer vision system 100 can be employed for administrator control. In particular, the deployed computer vision system 100 can be used for parental device control. In some cases, parents desire to restrict certain types of content from being viewed by their children on a user device. In this case, the system can allow the parents to customize the set of inputs represented by the set of signature(s) 132, e.g.. to represent specific topics or visual elements that parents wish to filter out from their child’s viewing. In this case, when a child attempts to access the content represented by the signatures, the deployed computer vision system 100 can process the content and modulate the output based on the parent-defined signatures using the modulation engine 130, thereby potentially blocking or altering the display of inappropriate material.Attorney Docket No. 56113-0839WO1

[0080]

[0065] As a further example, the deployed computer vision system 100 can be deployed in an autonomous vehicle or a robot to enhance system performance. For example, the base machine learning model 120 can be an object detection model that is used to detect objects as part of predicting the motion trajectory of the vehicle or robot, e.g., in response to detecting nearby objects. In this case, the set of signature(s) can include objects that were not included in the training data set used to train the base machine learning model 120, e.g., a new kind of scooter with wings that needs to be detected on the street, but which the deployed base machine learning model 120 has consistently classified as a seagull and ignored when motion planning.

[0081]

[0066] As illustrated in FIG. 2 A and FIG. 2B, the deployed computer vision system 100 can transition between tw o operational states based on the type of information, e.g., either a set of signature(s) 210 or resolution data 215 that indicates that the base machine learning model has been updated to achieve better performance on the sets of inputs represented by the previously received signatures, provided by another system, e.g., a server 200. In the particular example depicted, the deployed computer vision system 100 is deployed on a user device 220, e.g., a smartphone, laptop, tablet, smartwatch, or intemet-of-things (loT) device, to perform local processing of inputs, and the system 100 receives the information from the server 200 via an over-the-air update. In this context, an over-the-air update refers to a data update delivered without direct access to the user device 220, e.g., via a wireless connection to the server 200.

[0082]

[0067] In some cases, as shown in FIG. 2A, the server 200 can provide a set of signatures 210 to the deployed computer vision system 100. In some cases, the system 100 can activate the modulation engine 130 upon receipt of the set of signature(s) 210. In other cases, the modulation engine 130 is already activated and the system 100 can modify the signatures currently maintained by the modulation engine, e.g., based on receiving an updated set of signature(s) 210. In either case, as described in FIG. 1, the activated modulation engine 130 can process new inputs using both the base machine learning model 120 and the auxiliary machine learning model 134 to generate a modulated output 145 that takes into account the received set of signature(s) 210, or a base output 140 in the case that the new input does not include content represented by the signature(s) 210.

[0083]

[0068] In other cases, as depicted in FIG. 2B, the server 200 can provide resolution data 215 to the deployed computer vision system 100. In this case, this resolution 215 data can indicate that an update to the base machine learning model has resolved performance degradation issues previously addressed by the signatures, e.g., the set of signature(s) 210. For example, in response to receiving the resolution data 215, the system 100 can deactivate the modulation engine 130. While not depicted here, in another case, the server 200 can provide an indicationAttorney Docket No. 56113-0839WO1

[0084] that the set of signature(s) is an empty set to the modulation engine 130, and the system 100 can deactivate the modulation engine 130. When inactive, the deployed computer vision system 100 can process inputs using the base machine learning model to generate a base output 140 without modulation.

[0085]

[0069] As an example, the deployed computer vision system 100 can receive an update to the base machine learning model along with the resolution data 215. In particular, the resolution data 215 can indicate that the received update resolves the performance degradation caused by the sets of inputs represented by the signature(s) 210. In this case, upon receiving the resolution data 210, the system 100 can designate the auxiliary' machine learning model as inactive.

[0086]

[0070] While not depicted, in another case, the deployed computer vision system 100 can subsequently receive another set of signatures. In response to receiving the new set of signatures, the system 100 can activate the auxiliary machine learning model for processing inputs using the new signatures, e.g., as is depicted in FIG. 2A. In this case, the activation can enable the deployed computer vision system 100 to address new' performance issues or vulnerabilities as represented by the new set of signatures, e g., without requiring updates to the base machine learning model.

[0087]

[0071] By transitioning between the two operational states depicted in FIGS. 2A and 2B, respectively, the deployed computer vision system 100 provides for the dynamic adjustment of processing new inputs based on the current state of the modulation engine 130. More specifically, the activation of the modulation engine 130 through the receipt of signature(s) from the server 200 can enable the deployed computer vision system 100 to quickly adapt to newly identified inputs that have been demonstrated to degrade a performance of the base machine learning model, e.g., by leveraging the signature-based processing capabilities of the activated modulation engine 130.

[0088]

[0072] The generation of signatures for the deployed computer vision system 100 can be accomplished through various methods.

[0089]

[0073] FIG. 3 illustrates an example for generating signature embeddings by processing a text description, a set of example images, or both using a multimodal vision-language model 320. In the particular example depicted, the signatures are signature embeddings that capture key characteristics of inputs that have been determined to degrade the performance of the base machine learning model. For example, the signature generation engine 150 of FIG.l can generate the signature embeddings using a vision-language model as depicted in FIG. 3.

[0090]

[0074] A vision-language model is a multimodal neural network that is configured to process and integrate both textual and visual information, e.g., in order to perform a task that involvesAttorney Docket No. 56113-0839WO1

[0091] both textual and visual input modalities. In particular, the vision-language model 320 can be a machine learning model trained on both textual and visual data, allowing the model 320 to understand and process information across both modalities in a joint representation space. More specifically, the vision-language model 320 can include a trained image encoder neural network, e.g., that is configured to generate a representation of features of the image in a joint text-image representation space, and a trained text encoder neural network, e g., that is configured to generate a representation of features of the text in the joint text-image representation space.

[0092]

[0075] For example, the vision-language model 320 can be implemented as a Flamingo model, e.g., as is described in Alayrac, J., et. al. “Flamingo: a Visual Language Model for Few-Shot Learning” (arXiv:2204.14198v2). As another example, the vision-language model 320 can be implemented as a PaLM-E model, e.g., as is described in Driess, D., et. al.

[0093] “PaLM-E: An Embodied Multimodal Language Model” (arXiv:2303.03378vl). As yet another example, the vision-language model 320 can be implemented as an ALIGN model, a CLIP (Contrastive Language-Image Pretraining) model, or a DINO (self-distillation with no labels) model.

[0094]

[0076] In some cases, the vision-language model 320 can be a pretrained vision-language model. In this case, the vision-language model can be employed without being trained specifically for signature generation.

[0095] 177] As an example, the vision-language model 320 can process a text description of the set of input(s) 305 to generate the signatures in a zero-shot approach, e g., without requiring specific image examples. In particular, the text description can provide a natural language explanation of the visual elements or characteristics that should be captured in the signature embedding. In the particular example depicted, the text description of the set of input(s) 305 "Human face with a reflective surface in the background" characterizes a potential vulnerability in a user authentication system.

[0096]

[0078] In this case, the text description of the set of input(s) 305 can be obtained, e.g., from a user or another system. In some cases, the ability to generate signatures from text descriptions 305 alone can allow for rapid creation and deployment of new signatures without the need for any image collection. For example, this approach can be useful to immediately address newly discovered vulnerabilities or performance issues by processing a user input text description of the set of input(s) 305, e.g., a user can characterize an input that recently resulted in degraded performance of the model and submit the text description for use in a signature.Atorney Docket No. 56113-0839WO1

[0097]

[0079] As another example, the vision-language model 320 can process a set of example image(s) 315 that include the content that has been determined to degrade the performance of the base machine learning model in a few-shot approach. In this case the set of example image(s) 315 includes an image of a face with a shiny reflective surface in the background. In this case, the set of example image(s) 315 can be obtained through various means, e.g., including selecting actual image inputs to the system 100 that have resulted in unexpected behavior, photoshopping or digitally manipulating images that represent the inputs, generating synthetic image(s), or some combination.

[0098]

[0080] In some cases, one or more of the images in the set of example image(s) 315 can be generated using a generative machine learning model 310. In this case, the generative machine learning model 310 can have any appropriate machine learning architecture, e.g.. a neural network, that can be configured to process an input to generate an image that visually represents the input. In particular, the generative machine learning model can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate ty pe (e.g., fully-connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0099]

[0081] In the particular example depicted, the generative machine learning model 310 is configured for text-to-image generation. In this case, a text description, e.g., the text description 305, can be processed as an input to the generative machine learning model 310 to generate synthetic images representing the concept described in the text description of the set of input(s) 305.

[0100]

[0082] For example, the generative machine learning model 310 can be a generative adversarial network or a variational autoencoder. As another example, the generative machine learning model 310 can be a diffusion neural network that is configured to sequentially refine an initial state representing the text description of the set of input(s) 305 through a sequence of transformations that add noise to a data sample to generate an output image.

[0101]

[0083] In some cases, the generative machine learning model 310 can be implemented as a latent diffusion model that includes an encoder block to map images to a lower-dimensional latent space before applying the sequence of transformations and a decoder block to map from the lower-dimensional latent space back into image space. As an example, the generative machine learning model 310 can be implemented as a U-Net that includes skip connections that allow the model 310 to combine both coarse features from the beginning of the sequence of transformations and fine features from the end of the sequence of transformations to improve the generated image quality.Attomey Docket No. 56113-0839WO1

[0102]

[0084] In the case that few actual images of the set of inputs that have been determined to degrade the performance of the base machine learning model are available, the synthetic images can be used to create more robust signatures that capture a wider range of visual variations of the text description of the set of input(s) 305. In particular, the generated example images can provide a more diverse set of visual representations that capture various aspects of the text description of the set of input(s) 305, e.g.. than is provided by actual example images. For example, in the case of the user authentication example depicted, the generative machine learning model 310 can generate synthetic images illustrating different human faces with various types of reflective surfaces in the background, e.g., mirrors, polished metal surfaces, or glossy screens, which might be difficult to amass through standard data gathering techniques.

[0103]

[0085] As yet another example, the signatures can be generated by processing a combination of the text description of the set of input(s) 305 and the set of example images 310 using the vision-language model 320. In particular, by generating the signatures using both the text description 305 and the set of example images 310, the generated signatures can be more precise and robust. In particular, the vision-language model 320 can process the text description of the set of input(s) 305 and the set of example image(s) 315 to generate a signature embedding in the joint text-image space, thereby enabling the representation of both the related textual and visual information in a unified format.

[0104]

[0086] Furthermore, in some cases, signature embeddings generated using the vision-language model 320 can allow for more effective comparison and analysis of inputs across modalities. More specifically, by generating a signature embedding in the joint text-image representation space, the signature embedding can capture the salient features of both the textual description and the visual examples, thereby creating a richer representation of the set of inputs represented by the signature and allowing for more nuanced comparisons between the new input and each signature embedding using the auxiliary machine learning model. For example, in the case of the user authentication example, the signature embedding in the joint text-image space can encode information about reflective surfaces in the set of example image(s) 315 as well as characteristics of potential reflective surfaces the background provided by the text description of the set of inputs 305.

[0105]

[0087] In some cases, the deployed computer vision system can use the same vision-language model used to generate input embeddings as the auxiliary machine learning model. In this case, the deployed computer vision system can generate embeddings in the same joint text-Atorney Docket No. 56113-0839WO1

[0106] image space, e.g., in order to facilitate the direct comparison of the input embedding to the signature embeddings.

[0107]

[0088] The flexibility in generating signature embeddings using one or more of a text description or a set of example images can allow for rapid response to emerging visual trends or newly identified unsupported content types across a wide range of inputs. By employing the various signature generation methods described above, the deployed computer vision system 100 can adapt to new types of visual content or objects without requiring extensive data collection or model retraining, thereby improving the overall robustness and flexibility of the deployed computer vision system to handle a wider range of inputs.

[0108]

[0089] FIG. 4 is a flow diagram of an example process 400 for modulating an output of a base machine learning model. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a deployed machine learning system, e.g., the deployed computer vision system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0109]

[0090] The system can maintain a set of one or more signature(s) (step 410). Each signature in the set can represent a respective set of inputs to a base machine learning model that have been determined to degrade the performance of the base machine learning model, e.g., by resulting in poor or unexpected model performance. For example, the signatures can be signature embeddings that represent the respective set of inputs. In particular, by maintaining these signatures, the system can identify and address unsupported inputs without requiring training or finetuning of the base machine learning model.

[0110]

[0091] In some cases, at least one signature in the set of signatures can have been generated using a vision-language machine learning model. In this case, the vision-language machine learning model can process one or more of a text description of a respective input that has been determined to degrade the performance of the base machine learning model, or a set of one or more example images characterizing the respective input that has been determined to degrade the performance of the base machine learning model to generate a signature. More specifically, by using a vision-language model, the system can generate signatures that capture both textual and visual characteristics of the set of inputs that have been determined to degrade the performance of the base machine learning model. In particular, generating signature embeddings using a vision-language model can improve the system's robustness to the unsupported inputs represented by the set of signatures, and, in some cases, can provide for the ability to identify problematic inputs across different modalities.Attorney Docket No. 56113-0839WO1

[0111]

[0092] Moreover, in some cases, the system can generate an example image for the set of one or more example images processed by the vision-language model, e.g., using a generative machine learning model. For example, the generative machine learning model can be conditioned on the text description of the respective input that has been determined to degrade the performance of the base machine learning model. In particular, using a generative machine learning model allows for the creation of synthetic examples without the need for extensive real-world data collection to amass a data set for each set of inputs that have been determined to degrade the performance of the base machine learning model.

[0112]

[0093] The system can receive a new input to the base machine learning model (step 420). For example, this new input can be any type of visual data that the base machine learning model is configured to process. As an example, the new input can include an image, e.g.. a digital photograph, or a video, e.g., a video stream.

[0113]

[0094] For each signature in the maintained set, the system can process the new input and the signature using an auxiliary machine learning model to generate a respective auxiliary output that indicates whether the new input belongs to the respective set of inputs represented by the signature (step 430). In some cases, the auxiliary machine learning model can be inactive in a default state of the system, and activated when the set of one or more signatures is initially obtained, e.g., from another system or a server. More specifically, the system can use the signature(s) to identify potentially unsupported inputs that are similar to the set of inputs represented by the maintained signature(s). which have resulted in poor or unexpected base machine learning model performance.

[0114]

[0095] As an example, in the case that each signature is a signature embedding, the system can determine a measure of discrepancy, e.g., a measure of distance, between an embedding of the new input and the signature embedding. The system can then determine whether the discrepancy satisfies a threshold discrepancy criterion. In the case that the discrepancy satisfies the threshold discrepancy criterion, the system can indicate that the new input should be modulated. In some cases, the threshold discrepancy criterion can be a tunable parameter, allowing for adjustment of the system's sensitivity to potentially unsupported inputs. In this case, the system can update the threshold discrepancy criterion, e.g., using a validation data set.

[0115]

[0096] In some cases, the auxiliary machine learning model can be implemented as a visionlanguage machine learning model, e.g., in the case that the signature embeddings were generated using a vision-language machine learning model. In this case, the system can process the signature embedding and the new input using a vision-language machine learning model,Atorney Docket No. 56113-0839WO1

[0116] e.g., with an instruction to identify whether the new input has content characterized by the signature. As an example, the vision-language machine learning model can generate a similarity score that characterizes whether the new input has content characterized by the signature. In particular, using the vision language machine learning model can enable more nuanced comparisons between new inputs and the maintained set of one or more signature(s).

[0117]

[0097] The system can then modulate the output of the base machine learning model based on the respective auxiliary outputs generated for the set of one or more signature(s) (step 440). The modulation of the base machine learning model's output can be implemented in any appropriate way. For example, the system can override the output of the base machine learning model using the output of the auxiliary' machine learning model. As another example, the system can combine the output of the base machine learning model with the output of the auxiliary machine learning model. As yet another example, the system can generate a new output of the base machine learning model using the output of the auxiliary machine learning model, e g., by providing the output of the auxiliary machine learning model to the base machine learning model with an instruction to condition the output of the base machine learning model.

[0118]

[0098] In the case that the system determines that the new input does not belong to the respective set of inputs represented by any of the set of one or more signature(s), the system can provide the base output as the final output, thereby maintaining the standard behavior of the base machine learning model for inputs that do not match any identified unsupported inputs represented by the signature(s).

[0119]

[0099] By modulating the base model's output, the system can adjust its response to potentially unsupported inputs without requiring retraining or fine-tuning of the base machine learning model. In particular, the system can modulate the output to address performance degradation of the base machine learning model for new inputs represented by the set of signatures without training or finetuning the base machine learning model.

[0120]

[0100] In some cases, when the system modulates the output of the base machine learning model for the new input, the system can provide an alert. As an example, the alert can be indicative of the new input belonging to the set of inputs represented by a particular signature. In some cases, the alert can support the further monitoring of unsupported inputs represented by the signature(s).

[0121]

[0101] In some cases, the system can receive resolution data that indicates that an update to the base machine learning model has resolved the performance degradation caused by the respective sets of inputs represented by the maintained set of signature(s). In this case, uponAttorney Docket No. 56113-0839WO1

[0122] receiving the resolution data, the system can designate the auxiliary machine learning model as inactive for processing of further inputs to the base machine learning model. In particular, the system can return to the state of providing base outputs from the base machine learning model in response to the identification that the base machine learning model has been updated to address the previously identified unsupported inputs represented by the signature(s).

[0123]

[0102] Relatedly, the system can also receive an updated set of one or more signatures, e.g., a new set of signatures. For example, in response to receiving this updated set of signatures, the system can (re)activate the auxiliary machine learning model for processing inputs and the updated set of signatures, e.g., in order to modulate the output of the base machine learning model. In particular, the system is able to activate the auxiliary machine learning model for use with the set of signature(s) to quickly respond to newly identified unsupported inputs without requiring updates to the base machine learning model.

[0124]

[0103] In some implementations, the system can be deployed on a user device, e.g., a smartphone, laptop, or tablet. In this case, the set of one or more signature(s) can be received on the user device, e.g., in an over-the-air update from a server, thereby allowing for rapid distribution of new signatures and enabling quick responses to newly identified unsupported inputs without requiring updates to the base machine learning model.

[0125]

[0104] By implementing the process 400, the system can maintain the performance of a trained base machine learning model, while rapidly adapting to new types of unsupported inputs that have been determined to degrade the performance of the base machine learning model. In particular, using signature(s) to modulate the output of the base machine learning model can enhance the system's robustness and flexibility in real-world applications where novel inputs or emerging issues can arise quickly and unexpectedly by allowing for rapid adaptation to new ty pes of unsupported inputs.

[0126]

[0105] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0127]

[0106] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in thisAttorney Docket No. 56113-0839WO1

[0128] specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0129]

[0107] The term '‘data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0130]

[0108] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0131]

[0109] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions.Attorney Docket No. 56113-0839WO1

[0132] Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine: in other cases, multiple engines can be installed and running on the same computer or computers.

[0133]

[0110] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry' and one or more programmed computers.

[0134]

[0111] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by. or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0135]

[0112] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory' devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0136]

[0113] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g.. a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the userAtorney Docket No. 56113-0839WO1

[0137] can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0138]

[0114] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0139]

[0115] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework.

[0140]

[0116] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e g., an application server, or that includes a front-end component, e.g.. a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g.. a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0141]

[0117] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0142]

[0118] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context

[0143] Z1Attorney Docket No. 56113-0839WO1

[0144] of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0145]

[0119] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0146]

[0120] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0147]

[0121] What is claimed is:

Claims

1. Attorney Docket No. 56113-0839WO1CLAIMS1. A computer-implemented method for modulating an output of a deployed machine learning system, the method comprising:maintaining a set of one or more signatures that each represent a respective set of inputs to a base machine learning model that have been determined to degrade a performance of the base machine learning model;receiving anew input to the base machine learning model;for each of the set of one or more signatures:processing the new input and the signature using an auxiliary machine learning model to generate a respective auxiliary output that indicates whether the new input belongs to the respective set of inputs represented by the signature; andmodulating an output of the base machine learning model for the new input based on the respective auxiliary outputs for the set of one or more signatures.

2. The method of claim 1, further comprising:receiving an update to the base machine learning model and resolution data indicating that the update resolves the performance degradation caused by the respective sets of inputs represented by the one or more signatures in the set; andin response to receiving the resolution data, designating the auxiliary machine learning model as inactive for processing of further inputs to the base machine learning model.

3. The method of claim 2, further comprising:receiving an updated set of one or more signatures; andin response to receiving the updated set of one or more signatures, activating the auxiliary machine learning model for processing inputs and the updated set of signatures.

4. The method of any one of claims 1-3, wherein modulating the output of the base machine learning model addresses performance degradation of the base machine learning model for new inputs represented by the set of signatures without training or finetuning the base machine learning model.Atorney Docket No. 56113-0839WO15. The method of any one of claims 1-4, wherein at least one signature in the set of signatures is generated by processing an input comprising one or more of (i) a text description of a respective input that has been determined to degrade the performance of the base machine learning model, or (ii) a set of one or more example images characterizing the respective input that has been determined to degrade the performance of the base machine learning model using a vision-language machine learning model.

6. The method of claim 5, further comprising:generating an example image for the set of one or more example images using a generative machine learning model, wherein the generative machine learning model is conditioned on the text description of the respective input that has been determined to degrade the performance of the base machine learning model.

7. The method of any one or claims 1-6, wherein modulating an output of the base machine learning model for the new input based on the respective auxiliary outputs for the set of one or more signatures comprises at least one of:overriding an output of the base machine learning model using the output of the auxiliary7machine learning model;combining the output of the base machine learning model with the output of the auxiliary machine learning model; orgenerating a new output of the base machine learning model by providing the output of the auxiliary' machine learning model to the base machine learning model with an instruction to condition the output of the base machine learning model using the output of the auxiliary machine learning model.

8. The method of claim 7, further comprising:in response to determining the respective auxiliary output for the new input does not belong to the respective set of inputs represented by any of the set of one or more signatures, providing the base output as output.

9. The method of any one of claims 1-8, wherein the signature is a signature embedding that represents the respective set of inputs represented by the signature, and wherein processing the new input and the signature using an auxiliary machine learning model to generate the auxiliary output comprises:Attorney Docket No. 56113-0839WO1determining a measure of discrepancy between an embedding of the new input and the signature embedding;determining whether the discrepancy satisfies a threshold discrepancy criterion; and in response to determining that the discrepancy satisfies the threshold discrepancy criterion, indicating that the new input should be modulated.

10. The method of claim 9, wherein the threshold discrepancy criterion is a tunable parameter.

11. The method of any one of claims 9-10, wherein the signature and the embedding of the new input are generated using a vision-language machine learning model.

12. The method of claim 11, further comprising processing the signature and the new input using a vision-language machine learning model.

13. The method of any one of claims 1-12, further comprising:in response to modulating the output of the base machine learning model for the new input, providing an alert indicative of the new input belonging to the set of inputs represented by a first signature.

14. The method of any one of claims 1-13, wherein the auxiliary machine learning model is activated in response to receiving the set of one or more signatures.

15. The method of any one of claims 1-14, wherein the deployed machine learning system is deployed on a user device, and further comprising receiving the set of one or more signatures on the user device in an over-the-air update from a server.

16. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:maintaining a set of one or more signatures that each represent a respective set of inputs to a base machine learning model that have been determined to degrade a performance of the base machine learning model;Attorney Docket No. 56113-0839WO1receiving anew input to the base machine learning model;for each of the set of one or more signatures:processing the new input and the signature using an auxiliary machine learning model to generate a respective auxiliary output that indicates whether the new input belongs to the respective set of inputs represented by the signature; andmodulating an output of the base machine learning model for the new input based on the respective auxiliary outputs for the set of one or more signatures.

17. The system of claim 16, further comprising:receiving an update to the base machine learning model and resolution data indicating that the update resolves the performance degradation caused by the respective sets of inputs represented by the one or more signatures in the set; andin response to receiving the resolution data, designating the auxiliary machine learning model as inactive for processing of further inputs to the base machine learning model.

18. The system of claim 17, further comprising:receiving an updated set of one or more signatures; andin response to receiving the updated set of one or more signatures, activating the auxiliary machine learning model for processing inputs and the updated set of signatures.

19. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform operations comprising:maintaining a set of one or more signatures that each represent a respective set of inputs to a base machine learning model that have been determined to degrade a performance of the base machine learning model;receiving anew input to the base machine learning model;for each of the set of one or more signatures:processing the new input and the signature using an auxiliary machine learning model to generate a respective auxiliary output that indicates whether the new input belongs to the respective set of inputs represented by the signature; andmodulating an output of the base machine learning model for the new input based on the respective auxiliary outputs for the set of one or more signatures.Attorney Docket No. 56113-0839WO120. The computer storage medium of claim 19, further comprising:receiving an update to the base machine learning model and resolution data indicating that the update resolves the performance degradation caused by the respective sets of inputs represented by the one or more signatures in the set; andin response to receiving the resolution data, designating the auxiliary machine learning model as inactive for processing of further inputs to the base machine learning model.

Images