Ai agent for persistent campaign optimization

Abstract

A method efficiently tests campaign configuration modifications with reduced risk of performance degradation. The method includes obtaining content sponsor data for content sponsor(s) associated with respective campaign(s), generating a prompt based at least in part on the content sponsor data, and generating candidate modifications at least in part by applying the prompt to a generative artificial intelligence (Al) model. The method also includes generating test configuration(s) that differ from the configuration(s) of the respective campaign(s) in accordance with a first candidate modification, and applying the test configuration(s) to a virtual test environment that is isolated from a production environment. The method also includes determining performance metric(s) associated with applying the test configuration(s) to the virtual test environment.

Description

PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 Al AGENT FOR PERSISTENT CAMPAIGN OPTIMIZATION FIELD OF TECHNOLOGY

[0001] The present disclosure relates to digital advertising campaign data, and more specifically, to techniques for persistently restructuring and / or otherwise optimizing such data over time to improve performance metrics.BACKGROUND

[0002] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventor(s), to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0003] Digital advertising has become a highly technical field requiring, inter alia, that advertisers arrange, maintain, and continuously seek to improve complex accounts.Typically, an advertiser arranges such an account by creating a number of different advertising “campaigns” each focusing on a different area where there is a marketing / advertising need. Within each campaign, the advertiser is responsible for making a great number of decisions that will affect performance, hopefully for the better.“Performance” in this context may refer to measurable parameters / metrics such as impression rates, click-through rates, and so on. Examples of these campaign-specific decisions may include identifying which digital assets (digital advertisements) to use for the campaign, which existing text assets (e.g., descriptions and / or headlines) to associate with which existing images or videos, which bidding strategies to apply, which keywords to bid upon, budget allocations, the organization of specific assets / ads into specific asset / ad groups, creative decisions (e.g., how to modify assets or generate new assets), desired audience characteristics, and / or other decisions. The sum total of these decisions at any given time (e.g., as entered / provided by the advertiser via a user interface) is reflected in the account data as a particular campaign configuration, which is applied in a real- world production environment — along with the configurations of other campaigns of other advertisers — to direct specific content items (advertisements) to specific users (e.g., potential customers) over time in a competitive advertisement marketplace.

[0004] Configuring a campaign is a very time-intensive process, and typically requires a great deal of expertise and experience in order to achieve high performance. This isPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 especially cumbersome for advertisers with complex campaigns, and / or with accounts that include many (e.g., hundreds or thousands of) campaigns. Moreover, to maintain or improve high performance, it is necessary to update the campaign configuration over time as market conditions (e.g., relevant audiences, product / service offerings, competitor bidding strategies, etc.) change. Further still, making such updates is fraught with risk, as the complexity of the systems involved can easily result in any given change causing worse performance (e.g., a lower impression and / or click- through rate) for unforeseen reasons.

[0005] Accordingly, there is a need to automate the updating of digital advertising campaign configurations in a manner that is likely to improve performance, while presenting little to no risk of degraded performance. Moreover, and especially for systems that include many and / or highly complex campaigns of many advertisers, there is a need to provide such automated updates using highly efficient processing techniques (i.e., intelligent use of limited processing resources).SUMMARY

[0006] Generally, the disclosed techniques use a generative artificial intelligence (Al) model (e.g., a large language model (LLM)) to generate hypotheses regarding potentially useful (i.e., performance improving) modifications to campaigns. The hypotheses are specific to a particular content sponsor (e.g., advertiser), or specific to a group of content sponsors (e.g., a group of advertisers with similar characteristics such as selling a similar product or service, and / or with similar campaign configurations, etc.). The content sponsor(s) are associated with respective campaign(s) having particular configurations, where a production environment operates to provide specific content items (e.g., advertisements) to specific users (e.g., potential customers) based at least in part on those production environment configurations. To generate the sponsor- specific or group- specific hypotheses / candidate modifications, the disclosed techniques generate a prompt for the generative Al model based at least in part on content sponsor data (e.g., data indicating a service or product associated with the content sponsor(s), and / or campaign configurations of the content sponsor(s)). The generative Al model may be trained on configurations for historical campaigns and the performance metrics associated with those historical campaigns, for example.

[0007] For each candidate modification to be tested, the disclosed techniques may generate one or more test configurations (e.g., one configuration for the only content sponsor, or onePATENT APPLICATION Attorney Docket No: 31730 / 308499-00 configuration per content sponsor in a group) that differ from the production environment configuration(s) of the content sponsor campaign(s) in accordance with the candidate modification. The disclosed techniques further apply the test configuration(s) to a virtual test environment that is isolated from the production environment, such that neither the operation of the virtual test environment nor the configurations of any campaigns within the virtual test environment directly can affect the configurations of any campaigns running within the production environment or any performance metrics resulting from the campaign configurations in the production environment.

[0008] The disclosed techniques also determine one or more performance metrics associated with applying the test configuration(s) to the virtual test environment (e.g., one performance metric such as impression rate for each configuration / content sponsor, or multiple performance metrics for each configuration / content sponsor). In some implementations and scenarios, the disclosed techniques determine that the performance metric(s) satisfy a performance criterion (e.g., exceed a predetermined threshold value), and in response request user confirmation(s) of (e.g., permission to apply) the selected test configuration(s). If a particular content sponsor provides confirmation, the disclosed techniques may responsively modify the production environment configuration of that content sponsor so as to match that content sponsor’s test configuration, and then apply the modified configuration to the production environment for real-time operation.

[0009] In some implementations, the disclosed techniques also prioritize the hypotheses (candidate modifications) for virtual environment testing based on their expected impacts. For example, the generative Al model (or a separate model) may rank the various candidate modifications based on their likelihood of providing a significant or substantial boost to performance, and / or based on their likelihood of adoption by the content sponsor(s). Based on the prioritization, in these implementations, the disclosed techniques may select a particular candidate modification for testing before one or more other candidate modifications are tested.

[0010] Advantageously, by using a generative Al model as described above and disclosed herein, and in particular by generating the prompt using data specific to a particular content sponsor or a particular content sponsor group, the disclosed techniques can generate hypotheses / candidate modifications that are more closely tailored to that individual content sponsor or content sponsor group, and therefore are more likely to improve performance ofPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 the campaign(s) of the content sponsor or content sponsor group. Moreover, by testing candidate campaign configurations in a virtual test environment that is isolated from the production environment, poorly performing variants can be ignored or discarded without risk to the real-world performance of content-sponsor campaigns. Thus, campaign performance is improved while reducing the risk of unintentional performance degradation.Implementations that require content sponsor confirmation can further reduce the risk of unintentional performance degradation.

[0011] Furthermore, in various implementations, one or more features facilitate more efficient use of, and / or broader and / or more balanced access to, the limited computing resources of the virtual test environment. For example, in implementations that prioritize candidate modifications / hypotheses based on their expected impacts, efficiency is improved by preventing the use of limited processing resources (or preventing the near-term use of limited processing resources) for modifications that are less likely to have a positive impact (and / or are less likely to have a very substantial positive impact, etc.). For example, the disclosed techniques may help ensure that campaign configuration modifications with higher upside potential (a larger performance boost), and / or with a higher likelihood of content sponsor adoption / approval, are tested first in the virtual test environment. This can reduce the need for more processing resources (for example by reducing the number of candidate modifications / hypotheses that are required to be tested before an effective candidate modification / hypothesis is determined) while increasing the likelihood that more substantial improvements to a content sponsor’s campaign will be implemented at an earlier time.

[0012] As another example, in some implementations, the disclosed techniques apply one or more criteria to determine whether particular content sponsors are to be grouped together, such that a single hypothesis / campaign modification can be tested on behalf of multiple content sponsors. For example, if traffic associated with certain content sponsors is expected to be (e.g., has historically been) very low, the disclosed techniques may segment those content sponsors into groups (e.g., based on similarities between the content sponsors and / or their campaign configurations) and jointly test one or more test configurations on behalf of each such group. This can lead to more efficient use of the computing resources of the virtual test environment by, for example, helping to ensure that the computing resources are not wasted generating performance outcomes / metrics that are based on statistically insignificant testing volumes, and therefore unreliable.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0013] Other disclosed techniques, such as prioritizing virtual environment testing for different content sponsors based on value parameters associated with those content sponsors (e.g., volume of a service used by each content sponsor, etc.) may also result in more efficient use of the computing resources of the virtual test environment.

[0014] In one aspect, a method for efficiently testing campaign configuration modifications with reduced risk of performance degradation includes obtaining, by one or more processors, content sponsor data for one or more content sponsors associated with one or more respective campaigns, where a production environment providing content items of the one or more content sponsors to specific users based at least in part on configurations of the one or more respective campaigns. The method also includes generating, by the one or more processors, a prompt based at least in part on the content sponsor data, and generating, by the one or more processors and at least in part by applying the prompt to a generative artificial intelligence (Al) model, a plurality of candidate modifications to the one or more respective campaigns. The method also includes generating, by the one or more processors, one or more test configurations that differ from the configurations of the one or more respective campaigns in accordance with a first candidate modification of the plurality of candidate modifications, and applying, by the one or more processors, the one or more test configurations to a virtual test environment that is isolated from the production environment. The method also includes determining, by the one or more processors, one or more performance metrics associated with applying the one or more test configurations to the virtual test environment.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1 is a block diagram of an example system in which techniques for persistent campaign optimization can be implemented.

[0016] FIG. 2 depicts the structure of an example configuration for a campaign in a content sponsor account.

[0017] FIG. 3 depicts an example system that supports production and virtual test environments.

[0018] FIG. 4 depicts an example process for training and using a generative Al model for persistent campaign optimization.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0019] FIG. 5 is a flow diagram of an example method for efficiently testing campaign configuration modifications with reduced risk of performance degradation.DETAILED DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 is a block diagram of an example system 100 in which techniques for persistent campaign optimization can be implemented. The example system 100 includes a computing system 102, a client device 104 (e.g., a computing device or system of a user / consumer), a content sponsor 106 (e.g., a computing device or system of an advertiser or other content sponsor entity), a production system 108, a virtual test system 109, and a network 110. The computing system 102 may be remote from the client device 104, content sponsor 106, production system 108, and / or virtual test system 109, and may be communicatively coupled to the client device 104, content sponsor 106, production system 108, and / or virtual test system 109 via the network 110.

[0021] The network 110 may be a single communication network (e.g., the Internet), and in some implementations also includes one or more additional networks. As just one example, the network 110 may include a cellular network, the Internet, and a server- side local area network (LAN). While FIG. 1 shows only a single client device 104 and content sponsor 106, it is understood that the computing system 102 may also be in communication with a number (e.g., thousands or millions) of other client devices and / or content sponsors that are generally similar to the client device 104 and / or content sponsor 106, respectively.

[0022] Generally, in some implementations, computing system 102 may support advertising services for content sponsors (e.g., advertisers) such as content sponsor 106, to facilitate the marketing of commercial products and / or services of the content sponsors. To this end, computing system 102 may provide an online interface for content sponsor 106 and others to set up and maintain their own content sponsor accounts (e.g., digital advertising accounts). Content sponsor accounts can include any suitable settings and / or parameters that the content sponsors can configure to manage, for example, their digital advertising efforts. For example, the online interface may enable content sponsor 106 to set up, within an account of content sponsor 106, a number of digital advertising campaigns (e.g., associated with different areas of the business of the content sponsor 106, or different product lines, etc.).

[0023] FIG. 2 depicts the structure of an example configuration for a campaign 210 in a content sponsor account 200, which may be, for example, an account associated with contentPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 sponsor 106. As seen in FIG. 2, the example account 200 includes M (M being an integer greater than zero) campaigns. Each campaign may be associated with a particular product or service, or a particular subset of products or services, for example. In the depicted implementation, the campaign 210 includes a number of keywords 220. The content sponsor 106 may initially select the keywords 220 based on expectations of which types of queries might be entered by users interested in particular products or services of content sponsor 106, for example, and may link those keywords (or particular groups of those keywords) to particular content items (e.g., digital advertisements in the form of text, images, videos, and / or audio), or to particular groups of content items, that content sponsor 106 wishes to use for the particular products or services. In the example campaign 210, various content items are arranged into groups 230-1 through 230- A (A being an integer greater than zero). For example, each of groups 230-1 through 230- A may be associated with a different product or service.

[0024] The example campaign 210 also includes one or more bidding strategy parameters 222 and one or more budget allocation parameters 224. The bidding strategy parameters 222 may include, for example, bid amounts for specific keywords, a parameter indicating whether dynamic (e.g., Al-assisted) bidding is allowed, and / or other parameters. The budget allocation parameters 224 may include, for example, one or more parameters controlling which advertising channels are to be used, a parameter controlling how budget is shared across campaigns (or across content item groups, etc.), and / or other parameters. In other implementations, the campaign 210 may include more, fewer, and / or different elements or parameters than shown in FIG. 2. Collectively, the settings of the campaign 210, including the set of keywords, the mapping / arrangement of keywords (and possibly keyword groups), the associations of keywords or keyword groups to content items (e.g., digital advertisements), the bidding strategy and / or budget allocation parameter settings, and so on constitute a configuration that is specific to campaign 210. The configuration of campaign 210, any other campaigns of account 200, and possibly other metadata or information associated with account 200 (e.g., account identifier, content sponsor name, etc.), may be stored in an account database 160 along with similar information for a number of other content sponsor accounts.

[0025] Returning now to FIG. 1, the production system 108 includes one or more computing devices (e.g., one or more servers, such as an ad exchange server and possibly other servers) that may be co-located and / or remotely located, and generally provides a real-PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 world production environment in which the content items of content sponsors are provided to specific users / client devices over time in accordance with the campaign configurations of various content sponsors and in accordance with suitable content selection procedures (e.g., digital ad auctions). For example, the production system 108 may support a competitive advertisement marketplace for numerous advertisers (content sponsors), publishers (e.g., website hosts, streaming content platforms, etc.), and users of client devices.

[0026] In some implementations, production system 108 selects content items to present to specific users / client devices by executing an auction procedure that is based on keyword bids and / or other factors. As just one example, a user of the client device 104 may access a search engine via a web page hosted by a server not shown in FIG. 1, or via a search engine application (e.g., mobile application) that was previously installed on the client device 104, and the production system 108 may include selected content items to the client device 104 responsive to a search query. The search engine may be an all-purpose search engine that identifies / provides web pages and / or other Internet content, such as, for example, a Google® Search engine. Alternatively, the search engine may be associated with the search functionality of any other web page or application, such as a video sharing platform (e.g., YouTube®), a service-finding platform, and so on.

[0027] The virtual test system 109 includes one or more computing devices (e.g., one or more servers, such as an ad exchange server and possibly other servers) that may be colocated and / or remotely located, and generally provides a virtual image of the production environment of production system 108. For example, virtual test system 109 may include an image / copy of some or all of the software components, algorithms, models, etc., implemented by production system 108 to select particular content items for particular users / client devices. Unlike production system 108, however, operation of virtual test system 109 does not result in content items being provided / sent to any users / client devices such as client device 104. For example, whereas production system 108 may send (or cause to be sent) a particular advertisement image or video to client device 104 when that image or video is selected in an auction implemented by production system 108, virtual test system 109 may (in the same set of circumstances) select the image or video for client device 104 and record its selection, but not actually send (or cause any other device to send) the image or video to client device 104.

[0028] To allow low risk testing of campaign configurations, virtual test system 109 and production system 108 are isolated from each other. In particular, operation of the virtual testPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 environment as implemented by virtual test system 109 does not directly affect the configurations of any campaigns set up to run within the production environment implemented by production system 108. Moreover, operation of the virtual test environment as implemented by virtual test system 109 does not directly affect the operation of the production environment itself, or directly affect the performance metrics resulting from the application of such campaign configurations to the production environment. In some implementations, the production environment described herein for production system 108 and / or the virtual test environment described herein for virtual test system 109 are implemented entirely, or in part, by computing system 102. For example, computing system 102 may implement both the production environment and the virtual test environment, using different (physically and / or logically) servers of computing system 102.

[0029] The client device 104 of FIG. 1 may be the device of any user (e.g., potential customer) who is or could potentially be an intended consumer of content items selected / provided by production system 108. The client device 104 may be or include any stationary, mobile, or portable computing device with wired and / or wireless communication capability (e.g., a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart wearable device such as smart glasses or a smart watch, a vehicle head unit computer, etc.). In the example implementation of FIG. 1, client device 104 includes a network interface 120, a processor 122, memory 124, and a display 126. Processor 122 may be a single processor (e.g., a central processing unit (CPU)), or may include a set of processors (e.g., multiple CPUs, or one or more CPUs and one or more graphics processing units (GPUs)).

[0030] Memory 124 includes one or more computer-readable, non-transitory storage units or devices, which may include persistent (e.g., hard disk) and / or non-persistent memory components. Memory 124 stores instructions that are executable on the processor 122 to perform various operations, including the instructions of various software applications and the data generated and / or used by such applications. In the example implementation of FIG.1, memory 124 stores at least an application 130. Generally, application 130 is executed by processor 122 to provide one or more user interfaces via display 126, where the user interface(s) may enable a user to, for example, enter and submit search queries and view (among other things) digital advertisements or other content items in response to the queries. As another example, the user interface(s) may enable the user to view digital advertisements or other content items within content slots of information resources (e.g., web pages). ForPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 example, the application 130 may be a web browser application or a dedicated mobile application.

[0031] Display 126 includes hardware, firmware, and / or software configured to enable a user to view visual outputs of client device 104, and may use any suitable display technology (e.g., LED, OLED, LCD, etc.). In some implementations, display 126 is incorporated in a touchscreen having both display and manual input capabilities. Moreover, in some implementations where client device 104 is a wearable device, display 126 is a transparent viewing component (e.g., lenses of smart glasses) with integrated electronic components. For example, display 126 may include micro-LED or OLED electronics embedded in lenses of smart glasses.

[0032] Network interface 120 includes hardware, firmware, and / or software configured to enable client device 104 to exchange electronic data with computing system 102 via network 110. For example, network interface 120 may include a cellular communication transceiver, a WiFi transceiver, and / or transceivers for one or more other wired and / or wireless communication technologies.

[0033] While FIG. 1 shows client device 104 as a single component, in some implementations subcomponents of client device 104 are divided among two or more userside devices. As just one example, a pair of smart glasses may include processor 122, memory 124, and display 126, while a smartphone may include another processing unit, another memory, another display, and network interface 120. The smart glasses (or smart helmet, etc.) may then communicate as needed with the smartphone (e.g., via Bluetooth) to enable the operations of client device 104 described herein.

[0034] The computing system 102 includes a network interface 140, a processor 142, and memory 144. The network interface 140 includes hardware, firmware, and / or software configured to enable the computing system 102 to exchange electronic data with the content sponsor 106 (and other, similar entities), and possibly client devices such as client device 104, via the network 110. For example, the network interface 140 may include a wired or wireless router and a modem. The processor 142 may be a single processor (e.g., a central processing unit (CPU)), or may include a set of processors (e.g., multiple CPUs, or one or more CPUs and one or more graphics processing units (GPUs)). Computing system 102 may be a single computing device (e.g., server) at a single location, or may include multiple, coordinating computing devices that are either co-located or remotely distributed.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0035] Memory 144 includes one or more computer-readable, non-transitory storage media (e.g., units or devices), and may include persistent and / or non-persistent memory components. Memory 144 stores the instructions of a campaign optimization agent 150, a selection component 154, and a grouping component 156, each of which can be executed by the processor 142. As used herein, terms such as “optimization” and “optimize” are used to generally refer to improvements or attempted / potential improvements (e.g., an improvement or attempted improvement to a particular performance metric), and do not necessarily reflect any theoretical maximum or best case with respect to such improvements. While campaign optimization agent 150, selection component 154, and grouping component 156 are generally shown and described as distinct components, it is understood that selection component 154 and / or grouping component 156 may be part(s) of the campaign optimization agent 150, and / or that campaign optimization agent 150, selection component 154, and / or grouping component 156 may each include a number of different applications, software modules, and so on. In some implementations, selection component 154 and / or grouping component 156 do not reside at computing system 102. For example, grouping component 156 may be stored in the memory of a server not shown in FIG. 1, and selection component 154 may be a part of the virtual test system 109. Moreover, it is understood that, in some implementations, memory 144 may also store other software components not discussed or shown herein.

[0036] Memory 144 also stores a generative artificial intelligence (Al) model 152, which is used by campaign optimization agent 150 to improve campaign performance as described further herein. The generative Al model 152 may be a large language model (LLM), such as a transformer-based model (e.g., a BERT or GPT model), for example. In some alternative implementations, generative Al model 152 is stored in the memory of a remote computing device or system, and campaign optimization agent 150 uses / accesses generative Al model 152 remotely (e.g., via an application programming interface (API)). Regardless of where the trained generative Al model 152 resides, generative Al model 152 may be trained and / or updated or finetuned (e.g., as discussed below in connection with FIG. 4) locally by computing system 102, or remotely by another computing device or system.

[0037] Generally, and as described in further detail below, campaign optimization agent 150 uses the generative Al model 152 to automate the generation of hypotheses with respect to different campaign configurations for content sponsors (e.g., content sponsor 106), prioritizes (e.g., ranks) the hypotheses according to their expected impact (e.g., potential performance boost and / or likelihood of adoption by the content sponsor(s)), and tests thePATENT APPLICATION Attorney Docket No: 31730 / 308499-00 hypotheses in the virtual test environment of virtual test system 109 in accordance with the prioritization (e.g., in rank order). In some implementations, prioritization is performed by selection component 154 (e.g., as discussed below). In still other implementations, hypotheses are not prioritized.

[0038] To help ensure that the reason for any given performance improvement (or degradation) is better understood, and to lower the risk of making multiple configuration changes that work against each other, the hypothesized test configurations may in some implementations each be a change to only a single setting of a campaign configuration of the content sponsor. Campaign optimization agent 150 may iteratively (e.g., periodically or according to some other automated schedule) perform this process for any given content sponsor or content sponsor group in order to continually refine that content sponsor’s campaign configuration, in a manner that seeks to continually improve the content sponsor’ s campaign performance according to one or more measurable performance metric(s). By testing campaign configurations in the virtual test environment rather than the production environment, and possibly also by requesting content sponsor confirmation (e.g., while showing performance metrics from the virtual testing to the content sponsors) before applying the tested configurations to the production environment (e.g., before modifying a campaign configuration stored in account database 160), the risk of degraded real-world campaign performance can be greatly reduced (e.g., relative to testing such hypotheses / configurations in the production environment).

[0039] The prioritization may be determined by generative Al model 152, or by a separate model stored in memory 144 and not shown in FIG. 1. By prioritizing according to expected impact, the computing system 102 provides a more efficient and effective use of the limited processing resources of the virtual test system 109. For example, the campaign optimization agent 150 may cause the virtual test system 109 to first apply the hypothesized test configuration having the best expected performance improvement, and next cause the virtual test system 109 to apply the hypothesized test configuration having the next-best expected performance improvement, etc.

[0040] Selection component 154 generally filters out and / or de-prioritizes particular hypotheses / campaign modifications in order to increase efficiency of the computing resources of the virtual test system 109, and / or to ensure broader access to the virtual test system 109 across a large set of content sponsors. To this end, selection component 154 mayPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 impose one or more limits on virtual environment testing for a given content sponsor and / or a given campaign, and check the current status of the content sponsor and / or campaign against such limits when determining whether to apply additional test configuration(s) to the virtual test environment. For example, selection component 154 may disallow more than X % of a single content sponsor’s or single campaign’s hypotheses / modifications at any given time to be tested using the virtual test system 109, and / or disallow more than Y % of computing resources of virtual test system 109 to be used for a single content sponsor’s or single campaign’s hypotheses / modifications at any given time. As another example, selection component 154 may disallow testing of any hypothesis / modification that is expected to (or could possibly, etc.) use more than X % of the content sponsor’s real-world budget (per day, or per quarter, etc.), in order to avoid what that content sponsor is likely to view as a substantial cost. In still other examples, selection component 154 enforces one or more performance-based terminating conditions to ensure more efficient operation of, and / or broader access to, the virtual test system 109. For example, selection component 154 may cease testing hypotheses / campaign modifications for a given content sponsor when the virtual test system 109 indicates that the last X modifications tested for that content sponsor (within a given time window, or from a particular ranked list of modifications for that content sponsor, etc.) provide less than some threshold amount of performance improvement (individually or on average, etc., depending on the implementation), where X is a predetermined integer greater than zero.

[0041] Additionally or alternatively, in some implementations, selection component 154 estimates or predicts an expected impact of each hypothesis / modification on the performance of a given campaign, and prioritizes testing of modifications across different content sponsors and / or content sponsor groups based on which modifications are expected to have a relatively substantial positive impact. In this manner, selection component 154 may make it still more likely that virtual test environment 109 operates primarily on hypotheses / modifications that are relatively likely to provide an improvement (and / or are relatively likely to provide a more substantial improvement), thereby enhancing efficient use of the computing resources of virtual test system 109. Selection component 154 may additionally or alternatively stop testing hypotheses / campaign modifications tested for a given content sponsor based upon a stopping condition such as when the virtual test system 109 indicates that a threshold amount of performance improvement has been achieved, thereby preventing testing of unnecessary hypotheses. Because hypotheses for testing are prioritized, it will be appreciated that thePATENT APPLICATION Attorney Docket No: 31730 / 308499-00 stopping condition is achieved more quickly such that fewer hypotheses are required to be tested. In some implementations, selection component 154 determines these expected impacts based on outputs of campaign optimization component 150 (e.g., outputs from the operations described above with respect to prioritizing hypotheses for a single content sponsor or a single content sponsor group).

[0042] Grouping component 156 generally segments content sponsors into groups in order to apply joint testing of hypotheses / modifications, where appropriate, and thus increase the overall throughput of the virtual test system 109 (e.g., as measured in terms of individual content sponsor campaign modifications tested per unit of time). In some implementations, grouping component 156 determines that certain content sponsors should be grouped when those content sponsors are sufficiently similar in one or more respects (e.g., sell a same general type of product and / or have similar campaign configurations). For example, grouping component 156 may determine to group content sponsors using any suitable clustering technique(s) (e.g., / .-means clustering, or calculating cosine similarity values and comparing the calculated values to a similarity threshold, etc.). Clustering techniques may be based on a vector space having dimensions corresponding to parameters of the content sponsors and / or their associated campaign configurations, for example.

[0043] Additionally or alternatively, grouping component 156 and / or selection component 154 may determine to group content sponsors that are relatively small entities according to one or more value parameters (e.g., parameters indicating whether the content sponsors are historically associated with a low amount of traffic, a low ad spend amount, or another suitable indicator of a volume of service used by the content sponsor.). This may help to ensure that virtual test environment 109 only operates on (or more likely or more often operates on) hypotheses / modifications in instances where the performance outcomes encompass enough events to be statistically significant. In some implementations, selection component 154 first identifies all participating content sponsors that are relatively small entities, after which grouping component 156 uses a suitable clustering or other technique to segment those smaller entities into one or more groups / clusters as discussed above.

[0044] FIG. 3 depicts an example system 300 that collectively supports production and virtual test environments using the production system 108 of FIG. 1 and the virtual test system 109 of FIG. 1, respectively. It is understood, however, that many other arrangements, components, inputs, etc., are also possible.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0045] In the example implementation of FIG. 3, the production system 108 includes a supply-side component 312, a demand-side component 314, and a content selection procedure (CSP) component 316. The supply-side component 312 interacts with a publisher platform 320 to, for example, identify content item placement / impression opportunities (e.g., content slots for image or video advertisements within web pages, or slots for video advertisements within primary video content, etc.). The publisher platform 320 may itself be a part of supply-side component 312, or may be separately implemented. The demand-side component 314 interacts with a content sponsor platform 322 to, for example, obtain initial campaign configuration settings (and / or manual modifications to such settings) and possibly content items (e.g., digital assets / creatives / advertisements) from content sponsors such as content sponsor 106. The content sponsor platform 322 may itself be a part of demand-side component 314, or may be separately implemented.

[0046] The CSP component 316 executes one or more content selection procedures to select particular content items to provide to particular users of client devices 326 (e.g., similar to client device 104) for particular impression opportunities (e.g., available slots / times / etc.). In some implementations, the content selection procedure is an auction procedure based on keyword bids and possibly also other factors. For example, a given configuration of campaign configurations 324 (e.g., the configuration of campaign 210) may indicate a content sponsor’s maximum bids for each keyword of a set of keywords (e.g., keywords 220), and the CSP component 316 may apply / enforce those maximum bids (and / or other bidding rules or settings) when that content sponsor or that content sponsor’ s content item is a participant in an auction.

[0047] Other factors considered by CSP component 316 when selecting content items for available opportunities may include, for example, relevancy of the content item to the user of a client device. To this end, the CSP component 316 may obtain one or more types of contextual signals (e.g., locations, query information, etc.) directly from client devices 326 and / or from one or more context servers 328, after the respective users of client devices 326 have agreed to make such information available to CSP component 316. The CSP component 316 or another component of production system 108 may use the contextual signals to generate scores for particular content items in particular contexts (e.g., using a neural network or other machine learning model), and use the scores in combination with content sponsor bidding strategy parameter settings (e.g. keyword bids) to select which content item to present to a user for a given opportunity. More generally, it is understood thatPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 CSP component 316 may implement any suitable content selection procedure(s) and / or factor(s) to select content items to be provided to client devices 326, and that each of one, some, or all of campaign configurations 324 may include one or more parameters or settings that affect the likelihood of particular content items being selected for presentation to particular client devices or users, and / or the circumstances in which particular content items are selected for presentation to particular client devices or users.

[0048] When selecting a particular content item for a particular impression opportunity, the production system 108 sends the selected content item to the appropriate one of client devices 326, or otherwise causes (e.g., via another computing system) the selected content item to be sent to that client device. The production system 108 may also detect and record events associated with operation of the production environment, such as impressions, user “clicks” on content items, and so on. The production system 108 may further compute / compile performance measurement statistics based on these events, such as impression rates, click-through rates, average cost per view, cost per thousand impressions, return on ad spend, and so on.

[0049] The virtual test system 109 of the example system 300 includes a supply-side component 332, a demand-side component 334, and a CSP component 336. The components 332, 334, and 336 may be images (e.g., exact replicas, or nearly exact replicas) of the components 312, 314, and 316, respectively, of the production system 108, to ensure identical (or substantially identical) outputs for a given set of inputs. Moreover, the supply-side component 332 may obtain the same external inputs available to the supply-side component 312 (e.g., data from publisher platform 320), and the CSP component 336 may obtain some of the same external inputs available to the CSP component 316 (e.g., data from client devices 326 and / or context servers 328). However, whereas the production system 108 operates on the real-world campaign configurations 324 of content sponsors, the virtual test system 109 operates on “test” campaign configurations 340. These test configurations 340 may be generated by campaign optimization agent 150 using the generative Al model 152, as discussed further below (e.g., in connection with FIG. 4).

[0050] When the virtual test system 109 (i.e., CSP component 336) selects a particular content item for a particular opportunity at a particular client device, the virtual test system 109 does not actually send the selected content item (or otherwise cause the selected content item to be sent) to that client device. In some implementations, for example, the virtual testPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 system 109 is not configured, and / or connected with other systems, in a manner that permits the virtual test system to send (or cause the sending of) any content items to any of client devices 326. However, the virtual test system 109 may detect and record events associated with operation of the virtual test environment, such as by recording the selection of a particular content item for a particular opportunity at a client device as an “impression” for that content item. The virtual test system 109 may further compute / compile performance measurement statistics based on these events, such as impression rates, cost per thousand impressions, cost per impression, and so on. The virtual test system 109 may be unable to collect certain metrics, however, as a result of content not being provided to real-world users / devices. For example, the virtual test system 109 may be unable to collect click-through rate metrics.

[0051] FIG. 4 depicts an example process 400 for training and using a generative Al model for persistent campaign optimization. The process 400 may be implemented by one or more components of system 100, such as campaign optimization agent 150 as implemented by processor 142, for example. For ease of reference, the operations of process 400 are primarily described with reference to computing system 102 and other components of FIG. 1.

[0052] At stage 410 of the example process 400, computing system 102 trains the generative Al model using historical data stored in a historical database 162. The historical data may include performance outcomes / metrics associated with the real-world application of numerous campaign configurations of different content sponsors to the production environment provided / supported by production system 108. In some implementations, the generative Al model is generative Al model 152, and for ease of explanation the description below refers to such an implementation. In some implementations, the training at stage 410 is a finetuning of the generative Al model 152 (e.g., LLM) that was initially trained on a larger corpus of more general data.

[0053] The historical data 412 (e.g., from historical database 162) includes historical campaign configurations of content sponsors (e.g., campaign configurations that were previously applied to the production environment), along with historical performance metrics that serve as labels for the historical campaign contributions or are pre-processed to conform to an appropriate label format. The historical campaign configurations may include configurations of campaigns arranged in a manner similar to campaign 210, for example.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 The historical performance metrics may include impression rates, click-through rates, and / or any other performance metric(s) that the generative Al model 152 might seek to improve.

[0054] At stage 420 of the process 400, campaign optimization agent 150 generates, and in some implementations prioritizes, a set of hypotheses for optimizing a particular content sponsor’s campaign. Alternatively, the hypotheses may be for optimizing the campaigns of a particular set / group of content sponsors who are similar in some way. For example, the process 400 may include an additional stage (e.g., between stages 410 and 420) in which grouping component 156 and / or selection component 154 determine a content sponsor group that consists of only a subset of all content sponsors, based at least in part on similarities between characteristics of the content sponsors and / or similarities between the campaign configurations of the content sponsors (e.g., using a clustering algorithm or other suitable technique), and stage 420 may then include generating a set of hypotheses to be applied (or potentially be applied) to the virtual test environment for each of the content sponsors in that group. While the process 400 is described (and FIG. 4 is shown) primarily with respect to an implementation in which the generated hypotheses are specific to a single content sponsor, it is understood that the principles set forth can be extended to a group of content sponsors.

[0055] In some implementations where the generative Al model 152 is an LLM, stage 420 includes generating a prompt for the LLM based on content sponsor data associated with the content sponsor. For example, the content sponsor data may include data indicative of a service or product associated with the content sponsor (and / or other content sponsor characteristics such as location, etc.), and / or the configuration (e.g., parameter settings) of a campaign of the content sponsor. Alternatively or additionally, the content sponsor data may include a historical record of past campaign configurations and / or associated performance outcomes / metrics for the content sponsor (e.g., from historical database 162). In some implementations, the computing system 102 also uses market data to generate the prompt. For example, the computing system 102 may generate a prompt that also includes, or refers or links to, current market information such as recent average bid amounts per keyword across advertisers, and / or any other suitable market information.

[0056] Prompt generation may include using the content sponsor data to populate fields or other portions of a prompt template. For example, stage 420 may include generating a prompt using a template that begins with the language “Tow are an advertising expert managing a search ads campaign for a company that sells... , ” specifies content sponsor dataPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 (e.g., the product or service type, location, etc., of the content sponsor), and instructs that at least one hypothesis (or a particular number of hypotheses) be generated for potential testing. Alternatively or additionally, the prompt generation operation may include using a second, different LLM to generate the prompt, using the content sponsor data as part of a prompt that is input to the second LLM.

[0057] The generative Al model 152 accepts the generated prompt as an input, and outputs the requested information, including at least the requested hypothesis or hypotheses for improving campaign performance. For example, the generative Al model 152, having been trained or finetuned on historical data 412, may use its knowledge of what has and has not improved performance in the past (or degraded performance in the past) to hypothesize changes to the content sponsor’ s campaign configuration that are relatively likely to provide a substantial performance improvement(s) of the sort recited in the prompt.

[0058] In some implementations, the generative Al model 152 can output any modification to a campaign configuration, including, without limitation, changes to bidding strategy parameters, budget allocation parameters, mappings of content items (e.g., ads / creatives) to content item groups (e.g., ad groups) or other rearrangements of campaign structures, keywords and / or other audience-identifying parameters, the content items themselves (e.g., changing a background color, or a more substantial change), and / or any other suitable type of modification, subject to the constraints necessarily imposed by the nature of the training data / techniques that were used to create the generative Al model 152. In other implementations, the generative Al model 152 is further constrained by the prompt itself (e.g., if the prompt specifically requests hypotheses only for certain types of campaign modifications).

[0059] In some implementations, the prompt is generated at stage 420 in a manner that causes the generative Al model 152 to prioritize the generated hypotheses. For example, the prompt may include a statement instructing the generative Al model 152 to rank the hypotheses according to the expected impact of each hypothesis (i.e., each campaign modification). The expected impact may be described in the prompt in any suitable manner, such as potential performance impacts (e.g., having the greatest likelihood of any performance increase, or having the greatest degree of potential performance increase, etc.) and / or potential adoption rates (e.g., having the greatest likelihood of being adopted by the content sponsor). In some implementations where potential adoption rates are considered, thePATENT APPLICATION Attorney Docket No: 31730 / 308499-00 historical data 412 used to train the generative Al model 152 additionally includes data indicative of Al-generated campaign modifications that content sponsors refused to adopt in the past.

[0060] In other implementations, campaign optimization agent 150 or selection component 154 prioritizes hypotheses without using generative Al model 152. For example, campaign optimization agent 150 or selection component 154 may apply the hypotheses output by generative Al model 152 to one or more predictive models (e.g., neural networks) that output predicted performance metrics, or may access a table indicating average historical performance increases for particular types of campaign configuration modifications, etc., and then rank the hypotheses accordingly. In still other implementations, as noted above, hypotheses are not prioritized.

[0061] Each hypothesis corresponds to a particular campaign modification relative to the content sponsor or content sponsor group for whom the hypothesis is generated. In implementations where the hypothesis is specific to a single content sponsor, for example, campaign optimization agent 150 may, for each hypothesis / modification, generate a respective test configuration (e.g., one of test configurations 340) that matches the campaign configuration of the content sponsor (e.g., one of campaign configurations 324) but for the modification as indicated by the hypothesis. In implementations where hypotheses are generated for a group of content sponsors, campaign optimization agent 150 may for each hypothesis / modification generate a set of respective test configurations for the different group members that match the campaign configurations of those group members but for the modification as indicated by the hypothesis. In some implementations and / or scenarios, the campaign modification specifies a rule (e.g., percentage or other relative value) that the campaign optimization agent 150 can adapt to each group member’s campaign configuration (e.g., “increase keyword bid amounts for keywords including ‘shoes’ by 10%”).

[0062] At stage 430, the campaign optimization agent 150 causes the virtual test system 109 to apply one or more of the generated test configurations to the virtual test environment, in accordance with the prioritization (if any) determined at stage 420. This may include the campaign optimization agent 150 or selection component 154 first applying to the virtual test environment (e.g., via an API, etc.) the test configuration having the best expected impact (e.g., best potential performance improvement, or best chance of at least some performance improvement, etc.), then applying to the virtual test environment the test configuration havingPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 the next-best expected impact, etc. In some implementations where the virtual test system 109 supports parallel virtual test environments, the campaign optimization agent 150 or selection component 154 may cause the virtual test system 109 to first operate on the set of X hypothesized test configurations having the best expected impacts, and next operate on the set of X hypothesized test configurations having the next-best expected impacts, etc., where X is the number of test configurations (greater than one) that can be simultaneously handled by the virtual test system 109.

[0063] As noted above, the virtual test system 109 may operate identically (or in a manner very similar) to the production system 108, albeit without directly affecting the production environment of any real- world campaign configurations, and without providing any content items (e.g., ads) to real-world users or client devices. Moreover, in some implementations and to support true A / B testing / comparisons, virtual test system 109 obtains and operates on the same real-time, real-world signals operated upon by the production system 108 (e.g., signals from client devices 326, context servers 328, publisher platform 320, content sponsor platform 322, and / or other sources). By applying those real-time, real-world signals to supply-side component 332, demand-side component 334, and CSP component 336, the virtual test system 109 can determine precisely when particular content items would have been selected for (and presented to) particular users or client devices in the production environment, had the test configurations instead been applied to the production environment. The virtual test system 109 may also record / log performance outcomes / metrics associated with the application of each test configuration (e.g., impressions and impression rates).

[0064] Also at stage 430, the campaign optimization agent 150 may determine which hypotheses / test configuration(s) increased performance in a manner sufficient to warrant a recommendation of the corresponding campaign modification(s) to the content sponsor. For example, the campaign optimization agent 150 may determine which test configurations resulted in a performance increase (e.g., impression rate increase) greater than some predetermined threshold (e.g., 0%, 3%, 10%, etc.). In some implementations, the campaign optimization agent 150 determines performance increases relative to performance in the production environment (applying the real- world campaign configuration of the content sponsor) during the same time period over which the test configuration is tested in the virtual test environment, to facilitate A / B testing. In some implementations, the campaign optimization agent 150 (or another agent, application, system, etc.) uses the performancePATENT APPLICATION Attorney Docket No: 31730 / 308499-00 results to further train or finetune the generative Al model 152 (e.g., using reinforcement learning techniques), as indicated by the feedback 432 in FIG. 4.

[0065] At stage 440, the campaign optimization agent 150 causes a computing system or device associated with the content sponsor to present (e.g., via a graphical user interface of a web browser or dedicated application) the campaign modifications that had a sufficient (e.g., at least a threshold) performance improvement or likelihood of performance improvement. The modifications may be presented in ranked order by performance improvement or likelihood of performance improvement, and may indicate to a user the performance improvement or likelihood of performance improvement, for example. In some implementations, the graphical user interface presents all tested campaign modifications (e.g., in ranked order), without omitting any that failed to satisfy some criteria / threshold. In some implementations where the campaign modifications can be changes to the content items themselves, the graphical user interface shows a copy of the modified content item(s) (e.g., next to the unmodified version of the content item(s)).

[0066] The graphical user interface (or another graphical user interface that a user can navigate to) may include an interactive control that enables the content sponsor to accept or reject any given campaign modification that is presented. For example, a content sponsor may accept a proposed campaign modification that reconfigures the arrangement of content items (e.g., ad creatives) into groups (e.g., ad groups) and showed a 6% decrease in cost per impression, but reject a proposed campaign modification that increases the impression rate by 2% but only by increasing the overall spend for that content sponsor. As another example, the content sponsor may accept a proposed campaign modification that changes a background color of an ad creative, but refuse a proposed campaign modification that changes a text descriptor associated with the ad creative. In some implementations, the campaign optimization agent 150 (or another agent, application, system, etc.) uses the acceptance / rejection results to further train or finetune the generative Al model 152 (e.g., using reinforcement learning techniques), as indicated by the feedback 442 in FIG. 4.

[0067] At stage 450, the campaign optimization agent 150 modifies the campaign configuration of the sponsor’s real- world campaign (e.g., one of campaign configurations 324) in accordance with the campaign modification(s) accepted by the content sponsor, resulting in a modified campaign configuration 452. For example, the campaign optimization agent 150 may modify the real-world campaign configuration to match an accepted testPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 configuration, or (if multiple modifications are accepted) to apply all the differences / deltas between the test configurations and the current real- world configuration to the real-world configuration. The campaign optimization agent 150 may apply the change(s) by modifying account / campaign data of the content sponsor stored in account database 160, for example. The production system 108 may then operate on the modified campaign configuration 452 in real- world operation.

[0068] In some implementations and / or scenarios, the content sponsor’s acceptance of a proposed campaign modification at stage 440 is for purposes of applying the modification (at stage 450) to the regular operation of the content sponsor’s campaign in the production environment. In other implementations and / or scenarios, however, the content sponsor’s acceptance of a proposed campaign modification at stage 440 is for purposes of applying the modification (at stage 450) to the content sponsor’s campaign in an experimental mode in the production environment. For example, the experimental mode may apply to only a small portion of the content sponsor’s traffic, impression opportunities, etc. In either regular or experimental operation, the content sponsor can further modify the campaign as desired for further optimization or experimentation. In some implementations, the graphical user interface that presented at stage 440 includes one or more interactive controls that enable a user to reject a given campaign modification, apply the campaign modification in an experimental mode, or apply the campaign modification in a regular / full mode.

[0069] The campaign optimization agent 150 may perform iterations of the process 400 on any suitable time schedule or other basis. In some implementations, for example, the campaign optimization agent 150 obtains a new set of hypotheses for each content sponsor (or each content sponsor group) once each day, or once each week, etc. In other implementations, the campaign optimization agent 150 obtains a new set of hypotheses for each content sponsor (or each content sponsor group) in response to a different type of trigger, such as a content sponsor requesting (via a graphical user interface control) automated campaign optimization for the content sponsor.

[0070] As discussed above in connection with FIG. 1, in some implementations, selection component 154 and / or grouping component 156 (and / or another suitable component or device) may operate to filter out, prioritize, or otherwise sequence and / or limit the hypotheses / campaign modifications that are tested in the virtual environment. Such operations may occur between the testing (or potential testing) of differentPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 hypotheses / modifications at stages 420 and 430, and / or between different iterations of the process 400 in its entirety, for example.

[0071] FIG. 5 is a flow diagram of an example method 500 for efficiently testing campaign configuration modifications with reduced risk of performance degradation. The method 500 may be performed by computing system 102 (e.g., campaign optimization agent 150 as implemented by processor 142, with or without selection component 154 and / or grouping component 156), for example, possibly in combination with virtual test system 109 and / or other components, platforms, or systems. While not necessarily a part of the method 500, the method 500 occurs in a context where a production environment (e.g., provided / supported by production system 108) provides content items of the one or more content sponsors to specific users based at least in part on configurations of the one or more respective campaigns (e.g., campaign configurations 324).

[0072] At block 502, content sponsor data is obtained for one or more content sponsors associated with one or more respective campaigns. The content sponsor data may be indicative of, for example, a service or product associated with the content sponsor(s), a location of the content sponsor(s), the configuration(s) of the respective campaign(s), and / or other suitable information.

[0073] At block 504, a prompt is generated based at least in part on the content sponsor data. The prompt instructs a generative Al model (e.g., generative Al model 152) to generate one or more campaign modifications (hypotheses).

[0074] At block 506, candidate modifications to the respective campaign(s) are generated at least in part by applying the prompt to the generative Al model.

[0075] In some implementations, block 506 includes prioritizing the candidate modifications based at least in part on expected impacts (e.g., performance improvements, likelihood of performance improvements, likelihood of adoption by the content sponsor(s), etc.). In some such implementations, the prompt generated at block 504 includes instructions to prioritize the campaign modifications. In other implementations, the method 500 prioritizes the candidate modifications by inputting the campaign modifications generated at block 506 to a prioritization machine learning model that is distinct from the generative Al model to which the prompt is applied. In implementations that include prioritization, block 506 may include selecting a first candidate modification of the plurality of candidate modifications based at least in part on the prioritizing. The selected modification may be onePATENT APPLICATION Attorney Docket No: 31730 / 308499-00 with a highest ranking, for example, or one of a set of multiple modifications within the X highest rankings (e.g., if the virtual test system 109 can operate on X test configurations simultaneously), etc.

[0076] At block 508, one or more test configurations are generated. The test configuration(s) differ from the configurations of the one or more respective campaigns in accordance with a first candidate modification of the campaign modifications generated at block 506. For example, if the first candidate modification is to add a particular set of one or more keywords to the campaign, the test configuration may be identical to the real-world campaign configuration of the content sponsor(s) except for the addition of the additional keyword(s).

[0077] At block 514, the one or more test configurations is / are applied to a virtual test environment that is isolated from the production environment (e.g., applied to a virtual test environment provided / supported by virtual test system 109).

[0078] At block 516, one or more performance metrics associated with applying the one or more test configurations to the virtual test environment is / are determined. For example, block 516 may include determining an impression rate associated with applying a particular test configuration to the virtual test environment.

[0079] The method 500 may include multiple iterations of some or all of the blocks depicted in FIG. 5, on any suitable time schedule or other basis. In some implementations, for example, the method 500 includes an iteration once each day, or once each week, etc. In other implementations, iterations of the method 500 occur in response to a different type of trigger, such as a content sponsor requesting (via a graphical user interface control) automated campaign optimization for the content sponsor.

[0080] In some implementations, the method 500 may further include one or more operations to filter out, prioritize, or otherwise sequence and / or limit the modifications that are tested in the virtual environment (e.g., as discussed above in connection with the selection component 154). Such operations may occur between iterations of the blocks 508-512 (e.g., as the candidate modifications generated at block 506 are applied in ranked order to the virtual test environment), and / or between different iterations of the method 500 in its entirety, for example.

[0081] The method 500 may include one or more additional blocks not shown in FIG. 5. For example, the method 500 may include an additional block (e.g., before blocks 502 and / orPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 504) in which the generative Al model is trained using a training dataset indicative of (i) historical configurations for historical campaigns and (ii) historical performance metrics associated with the historical configurations. As another example, the method 500 may include an additional block in which the one or more performance metrics is / are caused to be displayed by one or more client devices (e.g., client devices similar to client device 104). As yet another example, the method 500 may include additional blocks in which (1) it is determined that a first performance metric of the performance metric(s) satisfies a performance criterion, (2) in response, content sponsor confirmation of the test configuration is requested, and (3) in response to receiving content sponsor confirmation of the test configuration, the campaign configuration is modified to match the test configuration and the modified configuration is applied to the production environment.

[0082] It is understood that the blocks of FIG. 5 need not be performed strictly in the order shown, and / or blocks of one iteration may occur in parallel with blocks of another iteration, etc.

[0083] In some implementations, the techniques disclosed herein use artificial intelligence to facilitate the restructuring of account data. Artificial intelligence (Al) is a segment of computer science that focuses on the creation of models that can perform tasks with little to no human intervention. Artificial intelligence systems can utilize, for example, machine learning, natural language processing, and computer vision. Machine learning, and its subsets, such as deep learning, focus on developing models that can infer outputs from data. The outputs can include, for example, predictions and / or classifications. Natural language processing focuses on analyzing and generating human language. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content, such as images, videos, text, audio, and / or other content, in response to input prompts and / or based on other information.

[0084] Example machine-learned models include neural networks or other multi-layer nonlinear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some machine-learned models can include multi-headed self-attention models (e.g., transformer models).PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0085] The model(s) can be trained using various training or learning techniques. The training can implement supervised learning, unsupervised learning, reinforcement learning, etc. The training can use techniques such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. A number of generalization techniques (e.g., weight decays, dropouts) can be used to improve the generalization capability of the models being trained.

[0086] The model(s) can be pre-trained before domain- specific alignment. For instance, a model can be pretrained over a general corpus of training data and fine-tuned on a more targeted corpus of training data. A model can be aligned using prompts that are designed to elicit domain- specific outputs. Prompts can be designed to include learned prompt values (e.g., soft prompts). The trained model(s) may be validated prior to their use using input data other than the training data, and may be further updated or refined during their use based on additional feedback / inputs.

[0087] In some implementations, the computing system 102 (e.g., account optimization agent 150) may use any one or more of the machine learning models noted above to perform one or more of the operations discussed herein in connection with machine learning. For example, the computing system 102 may use one such machine learning model (e.g., an LLM) as generative Al model 152, use one or more other of the machine learning models to predict expected impacts (e.g., predict performance resulting from, and / or adoption of, a given campaign modification), and so on.

[0088] Although the foregoing text sets forth a detailed description of numerous different aspects and implementations of the invention, it should be understood that the scope of the patent is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only and does not describe every possible implementation because describing every possible implementation would be impractical, if not impossible. Numerous alternative implementations could be implemented, using either current technology or technology developed after the filing date of this patent, which wouldPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 still fall within the scope of the claims. The disclosure herein contemplates at least the following examples:

[0089] Example 1. A method for efficiently testing campaign configuration modifications with reduced risk of performance degradation, the method comprising: obtaining, by one or more processors, content sponsor data for one or more content sponsors associated with one or more respective campaigns, wherein a production environment provides content items of the one or more content sponsors to specific users based at least in part on configurations of the one or more respective campaigns; generating, by the one or more processors, a prompt based at least in part on the content sponsor data; generating, by the one or more processors and at least in part by applying the prompt to a generative artificial intelligence (Al) model, a plurality of candidate modifications to the one or more respective campaigns; generating, by the one or more processors, one or more test configurations that differ from the configurations of the one or more respective campaigns in accordance with a first candidate modification of the plurality of candidate modifications; applying, by the one or more processors, the one or more test configurations to a virtual test environment that is isolated from the production environment; and determining, by the one or more processors, one or more performance metrics associated with applying the one or more test configurations to the virtual test environment.

[0090] Example 2. The method of Example 1, further comprising: prioritizing, by the one or more processors, the plurality of candidate modifications based at least in part on expected impacts of the plurality of candidate modifications; and selecting, by the one or more processors and based at least in part on the prioritizing, the first candidate modification, from among the plurality of candidate modifications, for testing in the virtual test environment.

[0091] Example 3. The method of Example 2, wherein generating the prompt includes adding to the prompt an instruction to indicate relative priorities for the plurality of candidate modifications, and wherein generating the plurality of candidate modifications includes the prioritizing.

[0092] Example 4. The method of Example 2 or 3, wherein prioritizing the plurality of candidate modifications based at least in part on expected impacts of the plurality of candidate modifications includes prioritizing the plurality of candidate modifications based on one or both of (i) potential performance impacts, and (ii) potential content sponsor adoption rates.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0093] Example 5. The method of any one of Examples 1-4, further comprising:prioritizing, by the one or more processors, campaign modifications of different content sponsors based at least in part on one or more value parameters associated with the different content sponsors.

[0094] Example 6. The method of Example 5, wherein the one or more value parameters associated with the different content sponsors include, for each content sponsor of the different content sponsors, a respective parameter indicative of a volume of service used by the content sponsor.

[0095] Example 7. The method of any one of Examples 1-6, further comprising: determining, by the one or more processors and based at least in part on the one or more performance metrics, not to apply one or more additional test configurations associated with the one or more content sponsors to the virtual test environment.

[0096] Example 8. The method of any one of Examples 1-7, wherein the content sponsor data consists of content sponsor data for a first content sponsor associated with a first campaign, and wherein the configurations of the one or more respective campaigns consist of a first configuration of the first campaign.

[0097] Example 9. The method of any one of Examples 1-7, wherein: the content sponsor data includes content sponsor data for a plurality of content sponsors associated with a plurality of respective campaigns; the configurations of the one or more respective campaigns include a plurality of configurations; and the method further comprises: determining, by the one or more processors, to jointly perform virtual environment testing for a content sponsor group that consists of the plurality of content sponsors, based at least in part on one or both of: one or more value parameters associated with the plurality of content sponsors; and one or both of (i) similarities between content sponsors in the plurality of content sponsors and (ii) similarities between configurations in the plurality of configurations

[0098] Example 10. The method of Example 9, wherein: determining to jointly perform virtual environment testing for the content sponsor group is based at least in part on the one or more value parameters; and the one or more value parameters include, for each content sponsor of the plurality of content sponsors, a respective parameter indicative of a volume of service used by the content sponsor.

[0099] Example 1 l.The method of Example 9, wherein: determining to jointly perform virtual environment testing for the content sponsor group is based at least in part on one orPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 both of (i) the similarities between content sponsors in the plurality of content sponsors and (ii) the similarities between configurations in the plurality of configurations.

[0100] Example 12. The method of any one of Examples 1-11, wherein generating the prompt based at least in part on the content sponsor data includes generating the prompt based at least in part on one or both of: a service or product associated with the one or more content sponsors; and the configurations of the one or more respective campaigns.

[0101] Example 13. The method of any one of Examples 1-12, further comprising: training, by the one or more processors, the generative Al model using a training dataset indicative of (i) historical configurations for historical campaigns and (ii) historical performance metrics associated with the historical configurations.

[0102] Example 14. The method of any one of Examples 1-13, wherein each configuration of the configurations of the one or more respective campaigns includes one or more campaign parameter settings, and wherein the plurality of candidate modifications includes a campaign parameter setting modification.

[0103] Example 15. The method of Example 14, wherein the campaign parameter setting modification includes a modification to one or both of: a bidding strategy parameter setting; and a budget allocation parameter setting.

[0104] Example 16. The method of any one of Examples 1-15, wherein each configuration of the configurations of the one or more respective campaigns includes a keyword set, and wherein the plurality of candidate modifications includes a keyword set modification.

[0105] Example 17. The method of any one of Examples 1-15, wherein each configuration of the configurations of the one or more respective campaigns includes a mapping of content items to content groups, and wherein the plurality of candidate modifications includes a mapping modification.

[0106] Example 18. The method of any one of Examples 1-15, wherein each configuration of the configurations of the one or more respective campaigns includes text associated with content items, and wherein the plurality of candidate modifications includes a text modification.

[0107] Example 19. The method of any one of Examples 1-18, wherein generating the prompt is further based on market data.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0108] Example 20. The method of any one of Examples 1-19, wherein the generative Al model is a large language model (LLM).

[0109] Example 21. The method of any one of Examples 1-20, wherein determining the one or more performance metrics includes: executing one or more content selection procedures in the virtual test environment; and generating the one or more performance metrics based at least in part on outcomes of the one or more content selection procedures.

[0110] Example 22. The method of any one of Examples 1-21, wherein determining the one or more performance metrics includes predicting the one or more performance metrics using a machine learning model.

[0111] Example 23. The method of any one of Examples 1-22, further comprising: causing, by the one or more processors, the one or more performance metrics to be displayed by one or more client devices.

[0112] Example 24. The method of any one of Examples 1-23, wherein the configurations of the one or more respective campaigns include a first configuration of a first campaign, the one or more test configurations include a first test configuration, and the method further comprises: determining, by the one or more processors, that a first performance metric of the one or more performance metrics satisfies a performance criterion; in response to determining that the first performance metric satisfies the performance criterion, requesting, by the one or more processors, content sponsor confirmation of the first test configuration; and in response to receiving content sponsor confirmation of the first test configuration, (i) modifying, by the one or more processors, the first configuration to match the first test configuration, and (ii) applying, by the one or more processors, the modified first configuration to the production environment.

[0113] Example 25. The method of any one of Examples 1-23, wherein the configurations of the one or more respective campaigns include a first configuration of a first campaign, the one or more test configurations include a first test configuration, and the method further comprises: determining, by the one or more processors, that a first performance metric of the one or more performance metrics satisfies a performance criterion; in response to determining that the first performance metric satisfies the performance criterion, requesting, by the one or more processors, user confirmation of the first test configuration; and in response to not receiving user confirmation of the first test configuration, using reinforcement learning to train or finetune the generative Al model.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00

[0114] Example 26. A system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of Examples 1-25.

[0115] Example 27. One or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of Examples 1-25.

[0116] The following additional considerations apply to the foregoing discussion.Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of the present disclosure.

[0117] Unless otherwise apparent from the context of use, reference in the present disclosure to a same set of “one or more processors” (or a same “plurality of processors,” etc.) performing multiple operations can encompass implementations in which performance of the operations is divided among the processor(s) in any suitable way. For example, “generating, by one or more processors, X; and generating, by the one or more processors, Y” can encompass: (1) implementations in which a first set of one or more processors (e.g., in a first computing device) generates X and an entirely distinct, second set of one or more processors (e.g., in a different, second computing device) independently generates Y; (2) implementations in which all processors in the set of one or more processors (e.g., all in the same device, or distributed among multiple devices) contribute to the generation of both X and Y; and (3) other variations.

[0118] Unless specifically stated otherwise, discussions in the present disclosure using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or aPATENT APPLICATION Attorney Docket No: 31730 / 308499-00 combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0119] As used in the present disclosure any reference to “one implementation” or “an implementation” means that a particular element, feature, structure, or characteristic described in connection with the implementation is included in at least one implementation or implementation. The appearances of the phrase “in one implementation” in various places in the specification are not necessarily all referring to the same implementation.

[0120] As used in the present disclosure, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0121] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs through the principles described herein. Thus, while particular implementations and applications have been illustrated and described, it is to be understood that the disclosed implementations are not limited to the precise construction and components disclosed in the present disclosure. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed in the present disclosure without departing from the spirit and scope defined in the appended claims.

Claims

1. PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 What is claimed is:

1. A method for efficiently testing campaign configuration modifications with reduced risk of performance degradation, the method comprising:obtaining, by one or more processors, content sponsor data for one or more content sponsors associated with one or more respective campaigns, wherein a production environment provides content items of the one or more content sponsors to specific users based at least in part on configurations of the one or more respective campaigns;generating, by the one or more processors, a prompt based at least in part on the content sponsor data;generating, by the one or more processors and at least in part by applying the prompt to a generative artificial intelligence (Al) model, a plurality of candidate modifications to the one or more respective campaigns;generating, by the one or more processors, one or more test configurations that differ from the configurations of the one or more respective campaigns in accordance with a first candidate modification of the plurality of candidate modifications;applying, by the one or more processors, the one or more test configurations to a virtual test environment that is isolated from the production environment; and determining, by the one or more processors, one or more performance metrics associated with applying the one or more test configurations to the virtual test environment.

2. The method of claim 1, further comprising:prioritizing, by the one or more processors, the plurality of candidate modifications based at least in part on expected impacts of the plurality of candidate modifications; and selecting, by the one or more processors and based at least in part on the prioritizing, the first candidate modification, from among the plurality of candidate modifications, for testing in the virtual test environment.

3. The method of claim 2, wherein generating the prompt includes adding to the prompt an instruction to indicate relative priorities for the plurality of candidate modifications, and wherein generating the plurality of candidate modifications includes the prioritizing.PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 4. The method of claim 2 or 3, wherein prioritizing the plurality of candidate modifications based at least in part on expected impacts of the plurality of candidate modifications includes prioritizing the plurality of candidate modifications based on one or both of (i) potential performance impacts, and (ii) potential content sponsor adoption rates.

5. The method of any one of claims 1-4, further comprising:prioritizing, by the one or more processors, campaign modifications of different content sponsors based at least in part on one or more value parameters associated with the different content sponsors.

6. The method of claim 5, wherein the one or more value parameters associated with the different content sponsors include, for each content sponsor of the different content sponsors, a respective parameter indicative of a volume of service used by the content sponsor.

7. The method of any one of claims 1-6, further comprising:determining, by the one or more processors and based at least in part on the one or more performance metrics, not to apply one or more additional test configurations associated with the one or more content sponsors to the virtual test environment.

8. The method of any one of claims 1-7, wherein the content sponsor data consists of content sponsor data for a first content sponsor associated with a first campaign, and wherein the configurations of the one or more respective campaigns consist of a first configuration of the first campaign.

9. The method of any one of claims 1-7, wherein:the content sponsor data includes content sponsor data for a plurality of content sponsors associated with a plurality of respective campaigns;the configurations of the one or more respective campaigns include a plurality of configurations; andthe method further comprises:PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 determining, by the one or more processors, to jointly perform virtual environment testing for a content sponsor group that consists of the plurality of content sponsors, based at least in part on one or both of:one or more value parameters associated with the plurality of content sponsors; andone or both of (i) similarities between content sponsors in the plurality of content sponsors and (ii) similarities between configurations in the plurality of configurations.

10. The method of claim 9, wherein:determining to jointly perform virtual environment testing for the content sponsor group is based at least in part on the one or more value parameters; andthe one or more value parameters include, for each content sponsor of the plurality of content sponsors, a respective parameter indicative of a volume of service used by the content sponsor.

11. The method of claim 9, wherein:determining to jointly perform virtual environment testing for the content sponsor group is based at least in part on one or both of (i) the similarities between content sponsors in the plurality of content sponsors and (ii) the similarities between configurations in the plurality of configurations.

12. The method of any one of claims 1-11, wherein generating the prompt based at least in part on the content sponsor data includes generating the prompt based at least in part on one or both of:a service or product associated with the one or more content sponsors; andthe configurations of the one or more respective campaigns.

13. The method of any one of claims 1-12, further comprising:training, by the one or more processors, the generative Al model using a training dataset indicative of (i) historical configurations for historical campaigns and (ii) historical performance metrics associated with the historical configurations.PATENT APPLICATION Attorney Docket No: 31730 / 308499-0014. The method of any one of claims 1-13, wherein each configuration of the configurations of the one or more respective campaigns includes one or more campaign parameter settings, and wherein the plurality of candidate modifications includes a campaign parameter setting modification.

15. The method of claim 14, wherein the campaign parameter setting modification includes a modification to one or both of:a bidding strategy parameter setting; anda budget allocation parameter setting.

16. The method of any one of claims 1-15, wherein each configuration of the configurations of the one or more respective campaigns includes a keyword set, and wherein the plurality of candidate modifications includes a keyword set modification.

17. The method of any one of claims 1-15, wherein each configuration of the configurations of the one or more respective campaigns includes a mapping of content items to content groups, and wherein the plurality of candidate modifications includes a mapping modification.

18. The method of any one of claims 1-15, wherein each configuration of the configurations of the one or more respective campaigns includes text associated with content items, and wherein the plurality of candidate modifications includes a text modification.

19. The method of any one of claims 1-18, wherein generating the prompt is further based on market data.

20. The method of any one of claims 1-19, wherein the generative Al model is a large language model (LLM).

21. The method of any one of claims 1-20, wherein determining the one or more performance metrics includes:PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 executing one or more content selection procedures in the virtual test environment; andgenerating the one or more performance metrics based at least in part on outcomes of the one or more content selection procedures.

22. The method of any one of claims 1-21, wherein determining the one or more performance metrics includes predicting the one or more performance metrics using a machine learning model.

23. The method of any one of claims 1-22, further comprising:causing, by the one or more processors, the one or more performance metrics to be displayed by one or more client devices.

24. The method of any one of claims 1-23, wherein the configurations of the one or more respective campaigns include a first configuration of a first campaign, the one or more test configurations include a first test configuration, and the method further comprises:determining, by the one or more processors, that a first performance metric of the one or more performance metrics satisfies a performance criterion;in response to determining that the first performance metric satisfies the performance criterion, requesting, by the one or more processors, content sponsor confirmation of the first test configuration; andin response to receiving content sponsor confirmation of the first test configuration, (i) modifying, by the one or more processors, the first configuration to match the first test configuration, and (ii) applying, by the one or more processors, the modified first configuration to the production environment.

25. The method of any one of claims 1-23, wherein the configurations of the one or more respective campaigns include a first configuration of a first campaign, the one or more test configurations include a first test configuration, and the method further comprises:determining, by the one or more processors, that a first performance metric of the one or more performance metrics satisfies a performance criterion;PATENT APPLICATION Attorney Docket No: 31730 / 308499-00 in response to determining that the first performance metric satisfies the performance criterion, requesting, by the one or more processors, user confirmation of the first test configuration; andin response to not receiving user confirmation of the first test configuration, using reinforcement learning to train or finetune the generative Al model.

26. A system comprising:one or more processors; andone or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1-25.

27. One or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1-25.

Images