Session evaluation API for message overhead reduction at scale

Abstract

Example embodiments of the present disclosure provide for an example method including accessing data associated with a number of viewing sessions associated with client devices. The method can include making an API call to a session evaluator for the viewing sessions and receiving a score for each viewing session. The method can include comparing the scores for the viewing sessions to a threshold score to determine whether to provide a one-to-one or cohort viewing session to the respective client device. For instance, scores exceeding the threshold score can be provided one-to-one viewing sessions and scores below the threshold score can be provided with cohort viewing sessions.

Description

SESSION EVALUATION API FOR MESSAGE OVERHEAD REDUCTION AT SCALEFIELD

[0001] The present disclosure relates generally to systems and methods for a session evaluation API for reducing message overhead at scale.BACKGROUND

[0002] Message delivery is utilized in real-time systems such as content serving during live viewing sessions. Existing systems make numerous calls for each content break within a live viewing session leading to large amounts of overhead in messages processed during live viewing sessions with large numbers of co-viewers.SUMMARY

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0004] In one example aspect, the present disclosure provides for an example system for session evaluation API for message overhead reduction at scale, including one or more processors and one or more memory' device storing instructions that are executable to cause the one or more processors to perform operations. In some implementations, the one or more memory devices can include one or more transitory or non-transitory computer-readable media storing instructions that are executable to cause the one or more processors to perform operations. In the example system, the operations can include accessing data associated with a plurality' of viewing sessions associated with a plurality of client devices. In the example system, the operations can include for each viewing session of the plurality of viewing sessions, making an API call to a session evaluator. In the example system, the operations can include obtaining, from the session evaluator, a plurality of scores for each respective viewing session of the plurality' of viewing sessions comprising a first score for a first viewing session and a second score for a second viewing session. In the example system, the operations can include comparing the first score and the second score to a threshold score. In the example system, the operations can include determining that the first score exceeds the threshold score. In the example system, the operations can include determining that the second score is below the threshold score. In the example system, the operations can include based on determining that the first score exceeds the threshold score and that the second score is below the threshold score: selecting the first viewing session to be a one-to-one contentserving session; and selecting the second viewing session to be a cohort serving session. In the example system, the operations can include transmitting one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data. In the example system, the operations can include transmitting cohort live stream data to a second device associated with the second viewing session and a plurality of additional devices comprising cohort serving session data.

[0005] In an example aspect, the present disclosure provides for an example computer-implemented method. The example method includes accessing data associated with a plurality of viewing sessions associated with a plurality of client devices. The example method includes for each viewing session of the plurality of viewing sessions, making an API call to a session evaluator. The example method includes obtaining, from the session evaluator, a plurality of scores for each respective viewing session of the plurality of viewing sessions comprising a first score for a first viewing session and a second score for a second viewing session. The example method includes comparing the first score and the second score to a threshold score. The example method includes determining that the first score exceeds the threshold score. The example method includes determining that the second score is below the threshold score. The example method includes based on determining that the first score exceeds the threshold score and that the second score is below the threshold score: selecting the first viewing session to be a one-to-one content serving session; and selecting the second viewing session to be a cohort serving session. The example method includes transmitting one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data. The example method includes transmitting cohort live stream data to a second device associated with the second viewing session and a plurality of additional devices comprising cohort serving session data.

[0006] In an example aspect, the present disclosure provides for an example transitory or non-transitory computer readable medium embodied in a computer-readable storage device and storing instructions that, when executed by a processor, cause the processor to perform operations. In the example transitory or non-transitory computer readable medium, the operations include accessing data associated with a plurality of viewing sessions associated with a plurality' of client devices. In the example transitory or non-transitory computer readable medium, the operations include for each viewing session of the plurality' of viewing sessions, making an API call to a session evaluator. In the example transitory or non-transitory computer readable medium, the operations include obtaining, from the session evaluator, a plurality of scores for each respective viewing session of the plurality of viewingsessions comprising a first score for a first viewing session and a second score for a second viewing session. In the example transitory or non-transitory computer readable medium, the operations include comparing the first score and the second score to a threshold score. In the example transitory or non-transitory computer readable medium, the operations include determining that the first score exceeds the threshold score. In the example transitory or non-transitory computer readable medium, the operations include determining that the second score is below the threshold score. In the example transitory or non-transitory computer readable medium, the operations include based on determining that the first score exceeds the threshold score and that the second score is below the threshold score: selecting the first viewing session to be a one-to-one content serving session; and selecting the second viewing session to be a cohort serving session. In the example transitory or non-transitory computer readable medium, the operations include transmitting one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data. In the example transitory or non-transitory computer readable medium, the operations include transmitting cohort live stream data to a second device associated with the second viewing session and a plurality of additional devices comprising cohort serving session data.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0008] Figure 1 depicts an example block diagram of an example system for a session evaluation API for message overhead reduction at scale according to example embodiments of the present disclosure.

[0009] Figure 2 depicts a flow chart diagram of an example method for a session evaluation API for message overhead reduction at scale according to example embodiments of the present disclosure.

[0010] Figure 3 depicts a block diagram of an example computing system for a session evaluation API for message overhead reduction at scale according to example embodiments of the present disclosure.DETAILED DESCRIPTION

[0011] Generally, the present disclosure is directed to generating session evaluation scores to determine whether live viewing sessions should be one-to-one content serving sessions or cohort serving sessions. Upon initiation of a viewing session by a client device, a call can bemade to a session evaluator API. The session evaluator API can facilitate the generation of a score associated with a percentile of the predicted value of the viewing session. Based on a comparison of the live viewing session’s score and a threshold score, the viewing session can be selected for either one-to-one content serving or cohort serving session.

[0012] Each live viewing session can include a number of breaks in the live content. Within each break (e.g., pod), there can be one or more content items provided for display via a user interface of a device associated with the live viewing session. In some cases, a call can be made during each break for personalized content items to be provided. However, in some cases, such as live viewing of sporting events, there can be hundreds of millions of live viewing sessions at the same time. As such, existing systems cannot serve personalized, e.g.. “one-to-one” content to every viewing session. A portion of the live viewing sessions can be selected for a cohort serving session. With a cohort serving session, a single stream of content can include pre-selected content items for the breaks of the live streaming content streamed to each of the devices associated with the cohort.

[0013] By intelligently selecting devices to be a part of the cohort viewing session versus the one-to-one viewing sessions, the number of calls made to a content item server during live viewing sessions can be reduced. As such, network traffic utilization and bandwidth utilization can be reduced, or even optimized for instance, through the adjustment of the threshold score. This can provide for improved network performance among many other benefits and technical advantages. The present disclosure additionally provides for improvements by intelligently selecting which devices to be provided one-to-one content serving and which devices to be grouped into a cohort serving session. As such, the present disclosure provides for a reliable determination (thanks to the use of the disclose session evaluation API, e.g., using a trained machine-learned model) of which devices to be grouped into a cohort serving session at the time of connection of the devices which was not possible with earlier methods. Thus, faster and more reliable live viewing sessions can be provided by publishers as a whole utilizing the session evaluation API. The use of the present API can advantageously adapt the distribution of content items (cohort vs. one-on-one) to the network conditions, number of viewers / connected devices or other parameters (as described hereafter) with each break in the live viewing session. For instance, a number of one-on-one sessions can be determined based for instance on the number of online view ers and network conditions, selected thanks to the API, the rest of the view ers being grouped in cohorts. Existing solutions would not bring such a finely tuned, yet adaptable, the distribution of thecontent items. Furthermore, the present disclosure provides for these improvements while maintaining, if not improving, system latency.

[0014] The improvements associated with the systems and methods discussed herein can be further understood with reference to the figures.

[0015] Figure 1 depicts a swim lane diagram of an example data flow 100 of data transfer between a publisher computing system 105, a session evaluator 110, and a content management service 115. In some instances, the session evaluator 110 can be associated with content management service 115. In some cases, the session evaluator 110 can be associated with the publisher computing system 105 and the content management service 115 can be associated with a content provider.

[0016] The publisher computing system 105 can transmit a session evaluator request 120 to session evaluator 110. The session evaluator request 120 can include user session signal data 125. User session signal data 125 can include data related to a current viewing session of a client device. In some cases, user session signal data 125 can include all available signals for a client device to make a determination of whether a one-to-one content serving session or a cohort serving session for a viewing session should be selected.

[0017] In some implementations, session evaluator request 120 can be an application programming interface (API) call made to the session evaluator 110. Session evaluator 110 can be an API. As described herein, an API can include a set of subroutine definitions, protocols, and / or tools that defines an interface between software components for the purpose of requesting or returning a data resource. An API can provide a set of clearly defined methods of communication between various software components. As such, a session evaluator API can be utilized to facilitate fulfilling session evaluator request 120 made to session evaluator 110.

[0018] Session evaluator 110 can receive session evaluator request 120 and process session evaluator request 120 to generate session evaluator response 130. Session evaluator response 130 can include session evaluation score 135. Session evaluator 110 can include one or more machine-learned models. The machine-learned models can obtain user session signal data 125. The machine-learned models can process user session signal data 125 and generate a session evaluation score 135 as output. In some instances, a first machine-learned model can generate an initial score which can be transformed into a score representing a percentile which can be transmitted as the session evaluation score 135. The percentile can represent an estimated value of a respective viewing session compared to all other expected or current viewing sessions occurring at the same time. In some instances, a percentile can berepresented as a number between 0 and 1. For instance, a score of 0.9 can be indicative of a viewing session expected to be more valuable than 90% of other viewing sessions occurring at the same time.

[0019] Publisher computing system 105 can receive session evaluator response 130.Publisher computing system 105 can process session evaluation score 135 to determine whether to provide the client device associated with the user session signal data 125 to determine whether to provide a one-to-one content item request or a cohort content item request for a respective live streaming session. For instance, publisher computing system 105 can determine the total number of live viewing sessions and determine a number or portion of the live viewing sessions that can be provided with a one-to-one request and a number or portion of the live viewing sessions that can be provided with a cohort request. For instance, the publisher computing system 105 can have a limit of 5 million viewing sessions which can be processed and served with one-to-one content items for the content item breaks within the respective live viewing sessions. As such, if there are 100 million live viewing sessions at the same time, the publisher computing system 105 can utilize the session evaluation score 135 for each respective viewing session to determine which 5 million viewing sessions to provide the one-to-one request and which viewing sessions to provide the cohort request. As such, the system can select 95 million viewing sessions for the cohort request.

[0020] The publisher computing system 105 can transmit the one-to-one content item request 140 to content management service 115. The one-to-one content item request 140 can be transmitted for each session selected by the publisher computing system 105 for the one-to-one request. By selecting a particular subset of live viewing sessions to provide a one-to-one request for. the publisher computing system 105 can reduce message overhead of the system.

[0021] For instance, without the present solution, the message overhead of the system would include messages requesting content items for each content item break within each respective live viewing session. The present disclosure allows the system to reduce message overhead by transmitting a single stream to each of the respective live viewing sessions associated with the cohort. As such, if a live viewing session includes 20 breaks over a 2 hour period, for each client device associated with the cohort, the number of requests being made to content management service 115 can be reduced from 20 messages (or more) to one single request for the cohort content items. As such, the strain on both the publisher computing system 105 and content management service 115 can be reduced.

[0022] It can be understood that more than one cohort can be generated. For instance, for a group of 90 millions live viewing sessions, several cohorts can be generated such that eachcohort is provided with different content items or different order of content items from the other cohorts. Alternatively, each of the live viewing sessions assigned to a cohort can be assigned to the same cohort and provided with the same live viewing stream.

[0023] In some instances, the session evaluator score that is generated can be calculated or otherwise adjusted based on the number of concurrent live viewing sessions (e.g., the number of live viewing sessions occurring in parallel). As such, when there is a larger number of sessions, there can be a larger number of cohorts or a larger number of sessions selected for cohorts. Additionally, or alternatively, when there is a smaller number of sessions, there can be a smaller number or percentage of sessions that are assigned to be a cohort viewing session. In some implementations, this can be determined based on an adjustment to the score threshold or an alteration to the machine-learned model.

[0024] Content management service 115 can process the one-to-one content item request 140 to provide content items to be provided to the client device associated with the user session signal data 125. The content management service 115 can utilize the user session signal data 125 to determine the content items to transmit to the client device. The content items can be streamed to the client device as audio and / or video content included within a live viewing session over a network. The client device can receive the content items and play the content items over the audio and / or video of the live viewing session.

[0025] Figure 2 depicts a flowchart of a method 200 to perform session evaluation score generation and utilization in selecting live streaming sessions for cohort viewing sessions or one-to-one viewing sessions according to example implementations of aspects of the present disclosure. The method 200 can be performed by processing logic that can include hardware (e.g., processing device, circuitry', dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, method 200 is performed by a server computing system (e.g., server computing system 330) or client computing system (e.g., client computing system 302). Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processors can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0026] At operation 202, processing logic can access data associated with a plurality of viewing sessions associated with a plurality of client devices. For instance, a publishersystem can provide a stream of content items to one or more devices via one or more network connections. In some instances, the stream of content items can include a live stream of content. By way of example, the live stream of content can be a sporting event live stream, such as a cricket match. In other instances, the live stream of content can include audio and / or video data associated with any live event or live content such as an interview.

[0027] The live streams or viewing sessions can be associated with live stream data which can include a number of predetermined content item breaks. Within each respective break of the predetermined content item breaks, there can be one or more content item slots. As such, within a content item break, there can be multiple content items served via the client device. This number of content items is increased based on the number of content item breaks within the live stream data.

[0028] In some implementations, the content item breaks can be dynamic breaks. As such, the timing of the breaks can vary based on the live stream data. In some instances, the first viewing session and the second viewing session can include a plurality of content item breaks.

[0029] In some embodiments, the live viewing session can include a viewing session on a connected television. In some embodiments, the live viewing session can include a viewing session on a mobile device. Additionally, or alternatively, the live viewing session can include a viewing session on a tablet computer, a desktop computer, a laptop computer, a smartphone, an on-board computer system, a networked computer appliance, a kiosk, wearable computer, or any device capable of streaming live content.

[0030] At operation 204, processing logic can, for each viewing session of the plurality of viewing sessions, make an API call to a session evaluator. At the time a live stream session initiates, the session evaluator API can be called by a client computing system. As such, processing logic can make a single call to a session evaluator API at the time the live stream session begins. In some implementations, a client device can exit out of a live stream session and re-initiate a session. In this example anew API call can be made to the session evaluator API to determine a score for the respective viewing session.

[0031] The API call to the session evaluator can include user session signal data associated with the viewing session. For instance, the session data can include at least one of: (i) a device type of the client device; (ii) a device identifier; (iii) a channel identifier; (iv) a network address associated with the client device; (v) a network type associated with the client device; (vi) a viewing session identifier; (vii) a content identifier associated with live viewing content; (viii) a time and date at which the live viewing content is scheduled to startand end; (ix) a category' identifier associated with the live viewing content; or (x) a content item identifier associated with the content item. In some implementations, the user session signal data used by the session evaluator can be the same signal data used by a content provider system in determining content items to be selected for one-to-one streaming sessions. By way of example, the user session signal data can include parameters such as device type, location, content type that are utilized in selecting and serving content items. As such, the signals can inform which live viewing sessions will have better value than others which can provide for more intelligent selection of live viewing sessions to be one-to-one content serving sessions as compared to being cohort serving sessions.

[0032] Additionally, or alternatively to the signals and parameters described above, the parameters can include an additional consent parameter, an all cue points parameter, a content source ID parameter, a video ID parameter, a correlator parameter, a description URL parameter, a device type hint parameter, an environment parameter, an ad manager schema indicator, a GDPR parameter indicative of whether the GDPR applies, a GDPR consent parameter, a video nonce parameter, a language parameter, an IAB exclusion URL parameter, a limited ads parameter, an application ID parameter, an application name parameter, a nonpersonalized ad parameter, a publisher privacy treatment parameter, a publisher provided identifier parameter, an application set ID parameter, an application set scope parameter, a resettable device identifier parameter, a restrict data processing parameter, a stream correlator parameter, an SDK API framework parameter, a session ID parameter, a server-side stitching source parameter, a traffic type parameter, a URL parameter, a video ad type parameter, a video duration parameter.

[0033] In some instances, session data can include a device ty pe, a software version associated with the device, a screen resolution, a screen size, a city or region location, an anticipated length of the session, a streaming service associated with the streaming session, or any other relevant session data.

[0034] At operation 206, processing logic can obtain, from the session evaluator, a plurality of scores for each respective viewing session of the plurality of viewing sessions comprising a first score for a first viewing session and a second score for a second viewing session. The session evaluator can receive session data for a respective client device viewing session and process the session data to generate a score. For instance, the session evaluator can generate the plurality' of scores based on session data associated with each viewing session of the plurality of viewing sessions.

[0035] In some implementations, the score can be a percentile indicative of a predicted value for the viewing session. For instance, a predicted value can include an average bid amount expected for the session or some other value associated with the content selection mechanisms. The predicted value can then be transformed into a more meaningful score by determining a distribution of the average bid amount or other predicted value for the entire set of live viewing sessions. As such, instead of having a random number with no context, the distribution or percentile can provide an actionable value to determine which viewing sessions to be selected for one-to-one versus cohort viewing sessions.

[0036] In some instances, the session evaluator can generate the plurality of scores using a machine-learned model. The machine-learned model can obtain user session signal data. The machine-learned model can process the user session signal data to generate a session evaluation score. In some instances, a first machine-learned model can generate an initial score which can be transformed into a score representing a percentile which can be transmitted as the session evaluation score. The percentile can represent an estimated value of a respective viewing session compared to all other expected or current viewing sessions occurring at the same time. In some instances, a percentile can be represented as a number between 0 and 1. For instance, a score of 0.9 can be indicative of a viewing session expected to be more valuable than 90% of other viewing sessions occurring at the same time.Additionally, or alternatively, a lower value can be indicative of a more valuable viewing session. For instance, a score of 0.1 can be indicative of viewing sessions expected to be in the top 10% of viewing sessions occurring at the same time.

[0037] At operation 208, processing logic can compare the first score and the second score to a threshold score. In some implementations, the threshold score can be determined or otherwise selected by the publisher computing system. In some implementations, the threshold score can be determined based on system resource constraints. For instance, the threshold score can be determined based on a capacity of the system to provide one-to-one content item serving sessions. In some instances, the threshold score can be a percentile set based on a total number of current or expected live viewing sessions. As such, based on the total number or current or expected live viewing sessions, the percentile can change to accommodate a set number of one-to-one content item requests.

[0038] At operation 210, processing logic can determine that the first score exceeds the threshold score. By determining that the first score exceeds the threshold score, processing logic can determine that a viewing session should be selected for a one-to-one content item serving session. For instance, if there are an expected number of 100 million active livestreaming sessions for a cricket match and the threshold number of one-to-one content item serving sessions that can be accommodated is 5 million, then the system can select viewing sessions associated with a score of 0.95 to 1.00 to be served one-to-one content items.

[0039] At operation 212, processing logic can determine that the second score is below the threshold score. Turning back to the 100 million active live streaming sessions with capacity to accommodate 5 million one-to-one content item serving requests, viewing sessions associated with a 0.00 to 0.94 score would be selected as a cohort serving session. As such, the method described herein can provide for more intelligent selection of which live streaming sessions to be associated with one-to-one versus cohort serving sessions and allocate resources accordingly.

[0040] At operation 214, based on determining that the first score exceeds the threshold score and that the second score is below the threshold score, processing logic can select the first viewing session to be a one-to-one content serving session and select the second viewing session to be a cohort serving session. As described herein, one-to-one content serving session data can include calls to a content provider for personalized, e.g., “one to one / ’ content items to be served for each break of the plurality of content item breaks. Cohort serving session data can include predetermined content items for each respective content item break of a serving session. As such, one-to-one content serving sessions can include realtime, or near real-time, determinations of content items to show during content item breaks. In some implementations, cohort content serving sessions can include a predetermined number of content items that are displayed in the same manner to every device associated with a cohort serving session.

[0041] At operation 216, processing logic can transmit one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data. Transmitting one-to-one live stream data to the first device can include transmitting live stream data to the first device that does not include a first content item for a first content item break. Upon the first device approaching a first content item break, the one-to-one live stream data can include a call to the content provider to provide content in real time, or near real time, to the first device. In some implementations, the first content item break can include a number of content item slots and content item requests. As such, the one-to-one live stream data transmitted to the first device can include a single call to the content provider to obtain and transmit content for the number of content item slots within the respective break.

[0042] At operation 218, processing logic can transmit cohort live stream data to a second device associated with the second viewing session and a plurality of additional devices comprising cohort serving session data. In some implementations the cohort serving session data can include predetermined content items for each respective content item slot of each content item break of the live stream session. As such, instead of making repeated calls to the content provider for each content item break, a single stream of content data can be transmitted to each device associated with the cohort. Thus, message overhead can be reduced by reducing the number of calls to the content provider during a live viewing session.

[0043] FIG. 3 depicts a block diagram of an example computing system 300 that performs low-latency file-based ordered message delivery at scale according to example embodiments of the present disclosure. The computing system 300 includes a client computing system 302, a server computing system 330, a training computing system 350, a session evaluator API 390, and a publisher computing system 370 that are communicatively coupled over a network 380.

[0044] The client computing system 302 can include a client device. A client device can be any type of computing device, such as, for example, a connected TV. television, set-top box. a personal computing device (e g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0045] The client computing system 302 includes one or more processors 312 and a memory 314. The one or more processors 312 can be any suitable processing device (e g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, and the like) and can be one processor or a plurality of processors that are operatively connected. The memory' 314 can include one or more transitory or non- transitory’ computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof. The memory 314 can store data 316 and instructions 318 which are executed by the processor 312 to cause the client computing system 302 to perform operations.

[0046] In some implementations, the client computing system 302 can store or include one or more machine-learned models 320. For example, the machine-learned models 320 can be or can otherwise include various machine-learned models such as Bayesian belief networks, kernel-based regularized least squares regression. Bay esian belief networks can include probabilistic graphical models comprising nodes and directed edges and learned from data. Kernel-based regularized least square regression can include non-linear regression estimation.In some implementations machine-learned models 320 can be or can otherwise include neural networks (e.g.. deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi -headed self-attention models (e.g., transformer models).

[0047] In some implementations, the one or more machine-learned models 320 can be received from the server computing system 330 over network 380, stored in the user computing device memory 314, and then used or otherwise implemented by the one or more processors 312. In some implementations, the client computing system 302 can implement multiple parallel instances of a single machine-learned model 320 (e.g., to perform parallel learning across multiple instances of content item selection).

[0048] More particularly, the overall model can include a suite of machine-learned modeling capabilities that are robust to data assumptions and easy to scale. The machine-learned modeling capabilities can determine a structure of a network (e.g., Bayesian belief network) and utilize the network for resource allocation determination. The output of the modeling capabilities can be used to more efficiently allocate resources to improve desired target outcomes. The machined learned models can be used for expediting the modeling process of media channel performance and allow for near real-time learnings of the effect of resource allocation on target actions performed.

[0049] Additionally, or alternatively, one or more machine-learned models 340 can be included in or otherwise stored and implemented by the server computing system 330 that communicates with the client computing system 302 according to a client-server relationship. For example, the machine-learned models 340 can be implemented by the server computing system 330 as a portion of a web service (e.g., a marketing service). Thus, one or more models 320 can be stored and implemented at the client computing system 302 and / or one or more models 340 can be stored and implemented at the server computing system 330.

[0050] The client computing system 302 can also include one or more user input components 322 that receives user input. For example, the user input component 322 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component canserve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0051] User computing system 302 can include one or more user interface(s) 324. For example, user interface(s) 324 can include graphical user interfaces, audio user interfaces, command line interfaces, menu-driven user interfaces, touch user interface, voice user interface, form-based user interface, or natural language user interfaces.

[0052] The server computing system 330 includes one or more processors 332 and a memory 334. The one or more processors 332 can be any suitable processing device (e g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, and the like) and can be one processor or a plurality of processors that are operatively connected. The memory 334 can include one or more transitory or non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof. The memory 334 can store data 336 and instructions 338 which are executed by the processor 332 to cause the server computing system 330 to perform operations.

[0053] In some implementations, the server computing system 330 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 330 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0054] As described above, the server computing system 330 can store or otherwise include one or more machine-learned models 340. For example, the models 340 can be or can otherwise include various machine-learned models. Example machine-learned models include can be or can otherwise include various machine-learned models such as Bayesian belief networks, kernel-based regularized least squares regression. Bayesian belief networks can include probabilistic graphical models comprising nodes and directed edges and learned from data. Kernel-based regularized least square regression can include non-linear regression estimation. In some implementations machine-learned models 340 can be or can otherwise include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0055] The client computing system 302 and / or the server computing system 330 can train the models 320 and / or 340 via interaction with the training computing system 350 that is communicatively coupled over the network 380. The training computing system 350 can be separate from the server computing system 330 or can be a portion of the server computing system 330.

[0056] The training computing system 350 includes one or more processors 352 and a memory 354. The one or more processors 352 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, and the like) and can be one processor or a plurality of processors that are operatively connected. The memory 354 can include one or more transitory or non-transitory computer-readable storage media, such as RAM, ROM. EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof. The memory 354 can store data 356 and instructions 358 which are executed by the processor 352 to cause the training computing system 350 to perform operations. In some implementations, the training computing system 350 includes or is otherwise implemented by one or more server computing devices.

[0057] The training computing system 350 can include a model trainer 360 that trains the machine-learned models 320 and / or 340 stored at the client computing system 302 and / or the server computing system 330 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0058] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 360 can perform a number of generalization techniques (e.g., weight decays, dropouts, and the like) to improve the generalization capability of the models being trained.

[0059] In particular, the model trainer 360 can train the machine-learned models 320 and / or 340 based on a set of training data 362. The training data 362 can include, for example, data associated with historical resource allocation (e.g., one or more months of media spend), daily data (e.g., resource allocation for various media channels, sales, conversions) at a geo level (e.g., zip code, city, or DMA level), weekly data (e.g.. resource allocation for various media channels, sales, conversions) at a geo level (e.g., zip code, city, or DMA level), ordifferent media weights across different time periods (e.g., when no resources are allocated to a media channel to correspond to an expected baseline). Media channels can include, for example, digital media, search, TV, and sales.

[0060] In some implementations, if the user has provided consent, the training examples can be provided by the client computing system 302. Thus, in such implementations, the model(s) 320 provided to the client computing system 302 can be trained by the training computing system 350 on user-specific data received from the client computing system 302. In some instances, this process can be referred to as personalizing the model.

[0061] The model trainer 360 includes computer logic utilized to provide desired functionality. The model trainer 360 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 360 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 360 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0062] The network 380 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 380 can be carried via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP. SMTP. FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0063] The machine-learned models described in this specification may be used in a variety of tasks, applications, and / or use cases.

[0064] In some implementations, the input to the machine-learned model(s) of the present disclosure can be statistical data. Statistical data can be, represent, or otherwise include data computed and / or calculated from some other data source. The machine-learned model(s) can process the statistical data to generate an output. As an example, the machine-learned model(s) can process the statistical data to generate a recognition output. As another example, the machine-learned model(s) can process the statistical data to generate a prediction output. As another example, the machine-learned model(s) can process the statistical data to generate a classification output. As another example, the machine-learned model(s) can process the statistical data to generate a segmentation output. As another example, the machine-learned model(s) can process the statistical data to generate a visualization output. As anotherexample, the machine-learned model(s) can process the statistical data to generate a diagnostic output.

[0065] In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, and the like). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.

[0066] In some implementations, the input to the machine-learned model(s) of the present disclosure can be sensor data. The machine-learned model(s) can process the sensor data to generate an output. As an example, the machine-learned model(s) can process the sensor data to generate a recognition output. As another example, the machine-learned model(s) can process the sensor data to generate a prediction output. As another example, the machine-learned model(s) can process the sensor data to generate a classification output. As another example, the machine-learned model(s) can process the sensor data to generate a segmentation output. As another example, the machine-learned model(s) can process the sensor data to generate a visualization output. As another example, the machine-learned model(s) can process the sensor data to generate a diagnostic output. As another example, the machine-learned model(s) can process the sensor data to generate a detection output.

[0067] FIG. 3 illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the client computing system 302 can include the model trainer 360 and the training data 362. In such implementations, the models 320 can be both trained and used locally at the client computing system 302. In some of such implementations, the clientcomputing system 302 can implement the model trainer 360 to personalize the models 320 based on user-specific data.

[0068] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken, and information sent to and from such systems. The inherent flexibility' of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0069] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, and equivalents.

[0070] The depicted and / or described steps are merely illustrative and can be omitted, combined, and / or performed in an order other than that depicted and / or described; the numbering of depicted steps is merely for ease of reference and does not imply any particular ordering is necessary or preferred.

[0071] The functions and / or steps described herein can be embodied in computer-usable data and / or computer-executable instructions, executed by one or more computers and / or other devices to perform one or more functions described herein. Generally, such data and / or instructions include routines, programs, objects, components, data structures, or the like that perform particular tasks and / or implement particular data t pes when executed by one or more processors in a computer and / or other data-processing device. The computer-executable instructions can be stored on a computer-readable medium such as a hard disk, optical disk, removable storage media, solid-state memory, read-only memory (ROM), random-access memory (RAM), or the like. As will be appreciated, the functionality of such instructions canbe combined and / or distributed as desired. In addition, the functionality can be embodied in whole or in part in firmware and / or hardware equivalents, such as integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or the like. Particular data structures can be used to implement one or more aspects of the disclosure more effectively, and such data structures are contemplated to be within the scope of computer-executable instructions and / or computer-usable data described herein.

[0072] Although not required, one of ordinary skill in the art will appreciate that various aspects described herein can be embodied as a method, system, apparatus, and / or one or more computer-readable media storing computer-executable instructions. Accordingly, aspects can take the form of an entirely hardware embodiment, an entirely software embodiment, an entirely firmware embodiment, and / or an embodiment combining software, hardware, and / or firmware aspects in any combination.

[0073] As described herein, the various methods and acts can be operative across one or more computing devices and / or networks. The functionality can be distributed in any manner or can be located in a single computing device (e.g., server, client computer, client device, or the like).

[0074] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications, and / or variations within the scope and spirit of the appended claims can occur to persons of ordinary skill in the art from a review of this disclosure. For example, one or ordinary skill in the art can appreciate that the steps depicted and / or described can be performed in other than the recited order and / or that one or more illustrated steps can be optional and / or combined. Any and all features in the following claims can be combined and / or rearranged in any way possible.

[0075] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and / or equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated and / or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, and / or equivalents

[0076] Figure 3 illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the client computing system 302 can include the model trainer 360 and the training data 362. In such implementations, the models 320 can be both trained and used locally at the client computing system 302. In some of such implementations, the client computing system 302 can implement the model trainer 360 to personalize the models 320 based on user-specific data.

[0077] The publisher computing system 370 includes one or more processors 372 and a memory' 374. The one or more processors 372 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, and the like) and can be one processor or a plurality of processors that are operatively connected. The memory 374 can include one or more transitory or non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof. The memory 374 can store data 376 and instructions 378 which are executed by the processor 372 to cause the publisher computing system 370 to perform operations.

[0078] In some implementations, the publisher computing system 370 includes or is otherwise implemented by one or more server computing devices. In instances in which the publisher computing system 370 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0079] As described above, the publisher computing system 370 can store or otherwise include one or more machine-learned models 379. For example, the models 379 can be or can otherwise include various machine-learned models. Example machine-learned models include can be or can otherwise include various machine-learned models such as Bayesian belief netw orks, kernel-based regularized least squares regression. Bayesian belief networks can include probabilistic graphical models comprising nodes and directed edges and learned from data. Kernel-based regularized least square regression can include non-linear regression estimation. In some implementations machine-learned models 379 can be or can otherwise include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural netw orks, recurrent neural netw orks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0080] The client computing system 302, the server computing system 330, and / or the publisher computing system 370 can train the models 320, 340. and / or 379 via interaction with the training computing system 350 that is communicatively coupled over the network 380. The training computing system 350 can be separate from the publisher computing system 370 or can be a portion of the publisher computing system 370.

[0081] The session evaluator API 390 can include one or more machine-learned model(s) 366. The session evaluator API 390 can receive session data and cause the one or more machine-learned model(s) 366 to evaluate one or more respective viewing sessions to generate a session evaluation score. The session evaluator API 390 can transmit session evaluation scores to the publisher computing system 370. The request pay load can include various user session parameters, such as device type, location, content type, and other relevant signals typically used in content item requests (e.g., ad requests). These parameters can be used by machine-learned model(s) 366 to generate a session score. The session score can be a normalized value between 0 and 1, representing the percentile of the session’s predicted monetization value compared to other sessions. The response payload can contain the session score and any relevant metadata. The API can be designed for low latency and high throughput to handle a large volume of requests. Error handling can be implemented to provide informative error messages in case of invalid requests or internal errors.Authentication and authorization mechanisms can be in place to secure access to the API.

[0082] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken, and information sent to and from such systems. The inherent fl exi bi 1 i t of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0083] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readilyapparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, and equivalents.

[0084] The depicted and / or described steps are merely illustrative and can be omitted, combined, and / or performed in an order other than that depicted and / or described; the numbering of depicted steps is merely for ease of reference and does not imply any particular ordering is necessary or preferred.

[0085] The functions and / or steps described herein can be embodied in computer-usable data and / or computer-executable instructions, executed by one or more computers and / or other devices to perform one or more functions described herein. Generally, such data and / or instructions include routines, programs, objects, components, data structures, or the like that perform particular tasks and / or implement particular data types when executed by one or more processors in a computer and / or other data-processing device. The computer-executable instructions can be stored on a computer-readable medium such as a hard disk, optical disk, removable storage media, solid-state memory, read-only memory (ROM), random-access memory (RAM), or the like. As will be appreciated, the functionality of such instructions can be combined and / or distributed as desired. In addition, the functionality can be embodied in whole or in part in firmware and / or hardware equivalents, such as integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs). or the like. Particular data structures can be used to implement one or more aspects of the disclosure more effectively, and such data structures are contemplated to be within the scope of computer-executable instructions and / or computer-usable data described herein.

[0086] Although not required, one of ordinary skill in the art will appreciate that various aspects described herein can be embodied as a method, system, apparatus, and / or one or more computer-readable media storing computer-executable instructions. Accordingly, aspects can take the form of an entirely hardware embodiment, an entirely software embodiment, an entirety firmware embodiment, and / or an embodiment combining software, hardware, and / or firmware aspects in any combination.

[0087] As described herein, the various methods and acts can be operative across one or more computing devices and / or networks. The functionality’ can be distributed in any manner or can be located in a single computing device (e.g., server, client computer, client device, or the like).

[0088] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications, and / or variations within the scope and spirit of the appended claims can occur to persons of ordinary skill in the art from a review of this disclosure. For example, one or ordinary skill in the art can appreciate that the steps depicted and / or described can be performed in other than the recited order and / or that one or more illustrated steps can be optional and / or combined. Any and all features in the following claims can be combined and / or rearranged in any way possible.

[0089] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and / or equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated and / or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, and / or equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising:accessing data associated with a plurality of viewing sessions associated with a plurality7of client devices;for each viewing session of the plurality of viewing sessions, making an API call to a session evaluator;obtaining, from the session evaluator, a plurality7of scores for each respective viewing session of the plurality7of viewing sessions comprising a first score for a first viewing session and a second score for a second viewing session;comparing the first score and the second score to a threshold score;determining that the first score exceeds the threshold score;determining that the second score is below the threshold score;based on determining that the first score exceeds the threshold score and that the second score is below the threshold score:selecting the first viewing session to be a one-to-one content serving session; andselecting the second viewing session to be a cohort serving session; transmitting one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data; andtransmitting cohort live stream data to a second device associated with the second viewing session and a plurality7of additional devices comprising cohort serving session data.

2. The computer-implemented method of claim 1, wherein the session evaluator generates the plurality of scores based on session data associated with each viewing session of the plurality^ of viewing sessions.

3. The method of claim 1, wherein the plurality of scores are a percentile indicative of a predicted value for the viewing session associated with a respective score.

4. The method of claim 1 , wherein the first viewing session and the second viewing session comprise a plurality of content item breaks.

5. The method of claim 4, wherein the one-to-one content serving session data includes calls to a content provider for personalized content items to be served for each break of the plurality of content item breaks.

6. The method of claim 1, wherein the cohort serving session data comprises predetermined content items for each respective content item break of a serving session.

7. The method of claim 1, wherein the session evaluator generates the plurality of scores using a machine-learned model.

8. The method of claim 1, wherein the plurality of viewing sessions comprise live viewing sessions on a connected television.

9. The method of claim 1, wherein the plurality7of viewing sessions comprise live viewing sessions on a mobile device.

10. A computing system, comprising:one or more processors; andone or more computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:receiving, by a session evaluation component, data associated with a plurality7of viewing sessions associated with a plurality of client devices;for each viewing session of the plurality of viewing sessions generating a first score for a first viewing session and a second score for a second viewing session;transmitting the first score and second score to a publisher system, wherein the publisher system performs operations comprising:comparing the first score and the second score to a threshold score; determining that the first score exceeds the threshold score;determining that the second score is below the threshold score; based on determining that the first score exceeds the threshold score and that the second score is below the threshold score:selecting the first viewing session to be a one-to-one content serving session; andselecting the second viewing session to be a cohort serving session;transmitting one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data; and transmitting cohort live stream data to a second device associated with the second viewing session and a plurality of additional devices comprising cohort serving session data.

11. The system of claim 10, wherein the session evaluator generates the plurality of scores based on session data associated with each viewing session of the plurality of viewing sessions.

12. The system of claim 11, wherein the session data comprises at least one of: (i) a device type of the client device; (ii) a device identifier; (iii) a channel identifier; (iv) a network address associated with the client device; (v) a network type associated with the client device; (vi) a viewing session identifier; (vii) a content identifier associated with live viewing content; (viii) a time and date at which the live viewing content is scheduled to start and end; (ix) a category identifier associated with the live viewing content; or (x) a content item identifier associated with the content item.

13. The system of claim 10, wherein the plurality of scores are a percentile indicative of a predicted value for the viewing session associated with a respective score.

14. The system of claim 10, wherein the first viewing session and the second viewing session comprise a plurality of content item breaks.

15. The system of claim 14, wherein the one-to-one content serving session data includes calls to a content provider for personalized content items to be served for each break of the plurality of content item breaks.

16. The system of claim 10, wherein the cohort serving session data comprises predetermined content items for each respective content item break of a serving session.

17. The system of claim 13, wherein the session evaluator generates the plurality of scores using a machine-learned model.

18. The system of claim 10, wherein the plurality of viewing sessions comprise live viewing sessions on a connected television.

19. The system of claim 10, wherein the plurality of viewing sessions comprise live viewing sessions on a mobile device.

20. One or more transitory or non-transitory computer readable media storing instructions that are executable by one or more processors to perform operations comprising:accessing data associated with a plurality of viewing sessions associated with a plurality of client devices;for each viewing session of the plurality of viewing sessions, making an API call to a session evaluator;obtaining, from the session evaluator, a plurality' of scores for each respective viewing session of the plurality of viewing sessions comprising a first score for a first viewing session and a second score for a second viewing session;comparing the first score and the second score to a threshold score; determining that the first score exceeds the threshold score;determining that the second score is below the threshold score;based on determining that the first score exceeds the threshold score and that the second score is below the threshold score:selecting the first viewing session to be a one-to-one content serving session; andselecting the second viewing session to be a cohort serving session; transmitting one-to-one live stream data to a first device associated with the first viewing session comprising one-to-one content serving session data; andtransmitting cohort live stream data to a second device associated with the second viewing session and a plurality of additional devices comprising cohort serving session data.

Images