Recommendation system observability
The method for recommendation system observability addresses the challenge of biased performance metrics by generating and evaluating content recommendations, enabling accurate measurement and improvement of content performance through incremental impact metrics.
Patent Information
- Application Number
- PCT/US2026/028220
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2026-05-08
- Filing Date
- 2026-05-14
- Publication Date
- 2026-11-19
AI Technical Summary
Conventional recommendation systems face challenges in effectively monitoring and measuring the performance of recommended content, leading to biased metrics and hindered debugging and user understanding of content effectiveness.
A method for recommendation system observability that generates and evaluates content recommendations, including explore and exploit types, to compute incremental impact metrics based on user feedback, enabling measurement and improvement of content performance.
Enables accurate measurement of content performance, allowing for debugging and re-training of machine learning models to improve recommendation systems and user understanding of content effectiveness.
Smart Images

Figure US2026028220_19112026_PF_FP_ABST
Abstract
Description
NETF0373PCRECOMMENDATION SYSTEM OBSERVABILITY CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of the United States Provisional Patent Application titled, “CONTINUOUS EXPLORE-EXPLOIT EVALUATION FRAMEWORK,” filed on May 15, 2025, and having Serial No 63 / 806,610 and claims the benefit of the United States Patent Application titled, “ RECOMMENDATION SYSTEM OBSERVABILITY,” filed on May 8, 2026, and having Serial No 19 / 672,264. The subject matter of these related applications is hereby incorporated herein by reference.BACKGROUNDTechnical Field
[0002] The embodiments of the present disclosure relate generally to computer science, streaming, and video processing technologies, and more specifically, to recommendation system observability.Description of the Related Art
[0003] Recommendation systems analyze user behavior, user preferences, and content attributes in order to recommend content that may be relevant to users. For example, a video streaming service can recommend media content to a user based on information such as media content previously watched by the user, media content liked or skipped by the user, media content watched by other similar users, and / or media content that is popular. One conventional approach for implementing a recommendation system includes training one or more machine learning models that predict relevant content for recommendation to users.
[0004] One drawback of the above approach involves the difficulty in monitoring and measuring the performance of content that are recommended by recommendation systems. Conventional statistics such as view counts, clicks, and likes, can be highly biased by the recommendation and presentation processes. For example, content that is heavily recommended by a recommendation system can receive more views, even when the incremental impact of such recommendations is less than the incremental impact of recommendations for other content. The inability to effectively monitor and measure the performance of content that is recommended by recommendation systems can hinder the debugging of the recommendationNETF0373PCsystems when those systems make ineffective or biased recommendations of content. The inability to effectively monitor and measure the performance of content recommended by recommendation systems also prevents users who create such content from understanding the effectiveness of the content.
[0005] As the foregoing illustrates, what is needed in the art are more effective techniques for recommendation system observability.SUMMARY
[0006] One embodiment of the present disclosure sets forth a computer-implemented method for recommendation system observability. The method includes generating, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations. The method also includes transmitting the one or more content recommendations to one or more endpoint devices. The method further includes receiving, from the one or more endpoint devices, feedback associated with user engagement with a first content recommended in the one or more content recommendations. In addition, the method includes computing, based at least on the feedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
[0007] Other embodiments of the present disclosure include, without limitation, one or more computer-readable media including instructions for performing one or more aspects of the disclosed techniques as well as one or more computing systems for performing one or more aspects of the disclosed techniques.
[0008] At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable the performance of content that is recommended by recommendation systems to be measured. The measured performance of content can be used to debug, and then to re-train or otherwise update, machine learning models used by the recommendation systems to generate improvement recommendations. The measured performance of content can also be output to improve user understanding of the effectiveness of content that was recommended. These technical advantages provide one or more technological improvements over prior art approaches.NETF0373PCBRIEF DESCRIPTION OF THE DRAWINGS
[0009] So that the manner in which the above recited features of the various embodiments can be understood in detail, a more particular description of the inventive concepts, briefly summarized above, may be had by reference to various embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of scope in any way, and that there are other equally effective embodiments.
[0010] FIG. 1 illustrates a network infrastructure used to distribute content to content servers and endpoint devices, according to various embodiments of the present disclosure.
[0011] FIG. 2 is a block diagram of a content server that can be implemented in conjunction with the network infrastructure of FIG. 1, according to various embodiments of the present disclosure.
[0012] FIG. 3 is a block diagram of a control server that can be implemented in conjunction with the network infrastructure of FIG. 1, according to various embodiments of the present disclosure.
[0013] FIG. 4 is a block diagram of an endpoint device that can be implemented in conjunction with the network infrastructure of FIG. 1, according to various embodiments of the present disclosure.
[0014] FIG. 5 illustrates a system for generating and evaluating content recommendations, according to various embodiments;
[0015] FIG. 6 sets forth a flow diagram of method steps for evaluating content recommendations, according to various embodiments; and
[0016] FIG. 7 sets forth a flow diagram of method steps for generating explore content recommendations, according to various embodiments.DETAILED DESCRIPTION
[0017] In the following description, numerous specific details are set forth to provide a more thorough understanding of the embodiments of the present disclosure.NETF0373PCHowever, it will be apparent to one of skill in the art that the embodiments of the present disclosure may be practiced without one or more of these specific details.
[0018] As described, one drawback of recommendation systems is the difficulty in monitoring and measuring the performance of content that is recommended by the recommendation systems. Conventional statistics such as view counts, clicks, and likes can be highly biased by the recommendation and presentation processes. For example, content that is heavily recommended by a recommendation system could receive more views, even when the incremental impact of each recommendation is less than the incremental impact associated with recommending other content. The inability to effectively monitor and measure the performance of content that is recommended by recommendation systems can hinder the debugging of the recommendation systems when those systems generate ineffective or biased recommendations. The inability to effectively monitor and measure the performance content that is recommended by recommendation systems also prevents users who create content from understanding the effectiveness of such content.
[0019] The disclosed techniques improve recommendation system observability. In some embodiments, a server application generates and provides, to playback applications, content recommendations that include (1) explore content recommendations used to determine the incremental impact of content being recommended, and (2) exploit content recommendations of content that is most relevant to users. The playback applications present the content recommendations to users and transmit user feedback, such as user selections, associated with the explore content recommendations to an evaluation application. The evaluation application computes, based on the user feedback, one or more incremental impact metrics that indicate the incremental impact of content in content recommendations. After computation, the incremental impact metric(s) can be used in any suitable manner, such as by being provided as feedback to users who generated content and / or being used to debug machine learning models that generate content recommendations.
[0020] Advantageously, the disclosed techniques enable the performance of content that is recommended by recommendation systems to be measured. The measured performance of content can be used to debug, and then to re-train or otherwise update, machine learning models used by the recommendation systems toNETF0373PCgenerate improvement recommendations. The measured performance of content can also be output to improve user understanding of the effectiveness of content that was recommended.System Overview
[0021] FIG. 1 illustrates a network infrastructure 100 used to distribute content to content servers 110 and endpoint devices 115, according to various embodiments of the present disclosure. As shown, network infrastructure 100 includes content servers 110, a control server 120, and endpoint devices 115, each of which are connected via a communications network 105.
[0022] Each endpoint device 115 communicates with one or more content servers 110 (also referred to as “caches” or “nodes”) via network 105 to download content, such as textual data, graphical data, audio data, video data, and other types of data. The downloadable content, also referred to herein as a “file,” is then presented to a user of one or more endpoint devices 115. In various embodiments, endpoint devices 115 may include computer systems, set top boxes, mobile computers, smartphones, tablets, console and handheld video game systems, digital video recorders (DVRs), DVD players, connected digital TVs, dedicated media streaming devices (e.g., the Roku® set-top box), and / or any other technically feasible computing platform that has network connectivity and is capable of presenting content, such as text, images, video, and / or audio content, to a user.
[0023] Each content server 110 may include a web-server, a database, and a server application 217 configured to communicate with control server 120 to determine the location and availability of various files that are tracked and managed by control server 120. Each content server 110 may further communicate with a fill source 130 and one or more other content servers 110 in order to “fill” each content server 110 with copies of various files. In addition, content servers 110 may respond to requests for files received from endpoint devices 115. The files may then be distributed from content servers 110 or via a broader content distribution network. In some embodiments, content servers 110 enable users to authenticate (e.g., using a username and password) in order to access files stored on content servers 110. Although only a single control server 120 is shown in FIG. 1, in various embodiments multiple control servers 120 may be implemented to track and manage files.NETF0373PC
[0024] In various embodiments, fill source 130 may include an online storage service (e.g., Amazon® Simple Storage Service, Google® Cloud Storage, etc.) in which a catalog of files, including thousands or millions of files, is stored and accessed in order to fill content servers 110. Although only a single fill source 130 is shown in FIG. 1, in various embodiments multiple fill sources 130 may be implemented to service requests for files. Further, as is well understood, any cloudbased services can be included in the architecture of FIG. 1 beyond fill source 130 to the extent desired or necessary.
[0025] FIG. 2 is a block diagram of a content server 110 that may be implemented in conjunction with network infrastructure 100 of FIG. 1 , according to various embodiments of the present disclosure. As shown, content server 110 includes, without limitation, a central processing unit (CPU) 204, a system disk 206, an input / output (I / O) device interface 208, a network interface 210, an interconnect 212, and a system memory 214.
[0026] CPU 204 is configured to retrieve and execute programming instructions, such as server application 217, stored in system memory 214. Similarly, CPU 204 is configured to store application data (e.g., software libraries) and retrieve application data from system memory 214. Interconnect 212 is configured to facilitate transmission of data, such as programming instructions and application data, between CPU 204, system disk 206, I / O device interface 208, network interface 210, and system memory 214. I / O device interface 208 is configured to receive input data from I / O devices 216 and transmit the input data to CPU 204 via interconnect 212. For example, I / O devices 216 may include one or more buttons, a keyboard, a mouse, and / or other input devices. I / O device interface 208 is further configured to receive output data from CPU 204 via interconnect 212 and to transmit the output data to I / O devices 216.
[0027] System disk 206 may include one or more hard disk drives, solid state storage devices, or similar storage devices. System disk 206 is configured to store non-volatile data such as files 218 (e.g., audio files, video files, subtitles, application files, software libraries, etc.). Files 218 can then be retrieved by one or more endpoint devices 115 via network 105. In some embodiments, network interface 210 is configured to operate in compliance with the Ethernet standard.NETF0373PC
[0028] System memory 214 includes a server application 217 and an evaluation application 220. Server application 217 is configured to service requests for files 218 received from endpoint device 115 and other content servers 110. When server application 217 receives a request for a file 218, server application 217 retrieves the corresponding file 218 from system disk 206 and transmits file 218 to an endpoint device 115 or a content server 110 via network 105. Evaluation application 220 is configured to compute, based on user feedback, one or more incremental impact metrics that indicate the incremental impact of including content in content recommendations, as discussed in greater detail below in conjunction with Figures 5-7.
[0029] FIG. 3 is a block diagram of a control server 120 that may be implemented in conjunction with network infrastructure 100 of FIG. 1 , according to various embodiments of the present disclosure. As shown, control server 120 includes, without limitation, a central processing unit (CPU) 304, a system disk 306, an input / output (I / O) devices interface 308, a network interface 310, an interconnect 312, and a system memory 314.
[0030] CPU 304 is configured to retrieve and execute programming instructions, such as a control application 317, stored in system memory 314. Similarly, CPU 304 is configured to store application data (e.g., software libraries) and retrieve application data from system memory 314 and a database 318 stored in system disk 306.Interconnect 312 is configured to facilitate transmission of data between CPU 304, system disk 306, I / O devices interface 308, network interface 310, and system memory 314. I / O devices interface 308 is configured to transmit input data and output data between I / O devices 316 and CPU 304 via interconnect 312. System disk 306 may include one or more hard disk drives, solid state storage devices, and similar storage devices. System disk 306 is configured to store a database 318 of information associated with content servers 110, fill source(s) 130, and files 218.
[0031] System memory 314 includes a control application 317 that is configured to access information stored in database 318 and process the information to determine the manner in which specific files 218 will be replicated across content servers 110 included in network infrastructure 100. Control application 317 may further be configured to receive and analyze performance characteristics associated with one or more of content servers 110 and / or endpoint devices 115.NETF0373PC
[0032] FIG. 4 is a block diagram of an endpoint device 115 that may be implemented in conjunction with network infrastructure 100 of FIG. 1, according to various embodiments of the present disclosure. As shown, endpoint device 115 may include, without limitation, a CPU 410, a graphics subsystem 412, an I / O device interface 414, a mass storage unit 416, a network interface 418, an interconnect 422, and a memory 430.
[0033] In some embodiments, CPU 410 is configured to retrieve and execute programming instructions stored in memory 430. Similarly, CPU 410 is configured to store and retrieve application data (e.g., software libraries) residing in memory 430. Interconnect 422 is configured to facilitate transmission of data, such as programming instructions and application data, between CPU 410, graphics subsystem 412, I / O device interface 414, mass storage unit 416, network interface 418, and memory 430.
[0034] In some embodiments, graphics subsystem 412 is configured to generate frames of video data and transmit the frames of video data to a display device 450. In some embodiments, graphics subsystem 412 may be integrated into an integrated circuit, along with CPU 410. Display device 450 may comprise any technically feasible means for generating an image for display. For example, display device 450 may be fabricated using liquid crystal display (LCD) technology, cathode-ray technology, and light-emitting diode (LED) display technology. An input / output (I / O) device interface 414 is configured to receive input data from user I / O devices 452 and transmit the input data to CPU 410 via interconnect 422. For example, user I / O devices 452 may comprise one or more buttons, a keyboard, and a mouse or other pointing device. I / O device interface 414 also includes an audio output unit configured to generate an electrical audio output signal. User I / O devices 452 include a speaker configured to generate an acoustic output in response to the electrical audio output signal. In alternative embodiments, display device 450 may include the speaker. A television is an example of a device known in the art that can display video frames and generate an acoustic output.
[0035] Mass storage unit 416, such as a hard disk drive or a flash memory storage drive, is configured to store non-volatile data. Network interface 418 is configured to transmit and receive packets of data via network 105. In some embodiments, network interface 418 is configured to communicate using the well-known Ethernet standard. Network interface 418 is coupled to CPU 410 via interconnect 422.NETF0373PC
[0036] In some embodiments, memory 430 includes programming instructions and application data that comprise an operating system 432, a user interface 434, and a playback application 436. Operating system 432 performs system management functions such as managing hardware devices including network interface 418, mass storage unit 416, I / O device interface 414, and graphics subsystem 412. Operating system 432 also provides process and memory management models for user interface 434 and playback application 436. User interface 434, such as a window and object metaphor, provides a mechanism for user interaction with endpoint device 115. Persons skilled in the art will recognize the various operating systems and user interfaces that are well-known in the art and suitable for incorporation into endpoint device 115.
[0037] In some embodiments, playback application 436 is configured to request and receive content from content server 110 via network interface 418. Further, playback application 436 is configured to interpret the content and present the content via display device 450 and / or user I / O devices 452.Recommendation system Observability
[0038] FIG. 5 illustrates a system 500 for generating and evaluating content recommendations, according to various embodiments. As shown, system 500 includes, without limitation, server application 217, playback application 436, and evaluation application 220. Server application 217 includes, without limitation, a recommendation system 506. Evaluation application 220 includes, without limitation, contents 524 and models 526 being evaluated.
[0039] In operation, server application 217 generates content recommendations 508, which include explore content recommendations 510 and exploit content recommendations 512. Server application 217 serves content recommendations 508 to playback applications 436, each of which executes on an endpoint device (e.q., endpoint device 115). Playback applications 436 present the content recommendations to users. Playback applications 436 receive user feedback 514 and transmit user feedback 514 to evaluation application 220. Evaluation application 220 computes, based on user feedback 514, one or more incremental impact metrics that indicate the incremental impact of including content 524 in content recommendationsNETF0373PC
[0040] Recommendation system 506 is a module of server application 217 that generates content recommendations 508. Each content recommendation 508 can include one or more content candidates 504. For example, when content candidates 504 include candidate rows of media content items, then each content recommendation 508 could include a page of row candidates. Content recommendations 508 include explore content recommendations 510 and exploit content recommendations 512. In some embodiments, recommendation system 506 determines a ranking of candidate content for each position on a page. Returning to the example of row candidates, recommendation system 506 could determine, using one or more trained machine learning models, a ranking of candidate rows based on contents of the candidate rows and user information, such that content predicted to be of interest to particular users is recommended to those users. For an exploit content recommendation 512, recommendation system 506 selects, fora position, the highest ranking candidate row that has not been used in any previous position. For an explore content recommendation 510, recommendation system 506 selects, for a position, a candidate row that has not previously been used based on a determined ranking for the candidate rows and a probability distribution. Accordingly, the highest ranking candidate row is not guaranteed to be selected. Instead, higher ranked rows have a higher probability of being selected according to the probability distribution. In some embodiments, the probability distribution indicates the probability with which each rank is selected, any suitable probability distribution can be used. For example, a Boltzmann distribution can be used in some embodiments. In some embodiments, the recommendation system 506 can generate explore content recommendations 510 and exploit content recommendations 512 based on a configurable frequency. For example, a certain percentage of content recommendations 508 (e.q., 10 percent) could be explore content recommendations 510, and the rest of content recommendations 508 could be exploit content recommendations 512.
[0041] Playback applications 436 execute on endpoint devices (e.q., endpoint device 115) that each receive one or more content recommendations 508. Each playback application 436 is configured to present the received content recommendations 508 via display and / or I / O devices (e.q., display device 450 and / or user I / O devices 452). Playback applications 436 can also interpret and present content items from a content server 110 via display device 450 and / or user I / ONETF0373PCdevices 452 in response to user selections of the content items in content recommendations 508.
[0042] In some embodiments, user feedback 514 includes user selections associated with content recommendations 508. Returning to the example of rows of content items, user feedback 514 could include selections of particular rows and / or content items therein. In some embodiments, user feedback 514 can also include other information, such as dwell time, engagement rate, abandonment rate, user retention rate, and / or other signals. User feedback 514 associated with explore 518 and exploit 520 content recommendations is included in training data 516 for training 522 (or re-training) machine learning model(s) used by recommendation system 506. The inclusion of user feedback 514 associated with explore 518 content recommendations allows the machine learning model(s) to learn from more diverse scenarios than if only user feedback 514 associated with exploit 520 content recommendations were used as training data 516. Training data 516 can also include any other suitable data. User feedback 514 is also provided to evaluation application 220 for evaluating contents 524 and models 526.
[0043] Evaluation application 220 is an application that evaluates contents 524 and models 526. Contents 524 can include content candidates 504 created by users 502. Models 526 can include one or more machine learning models used by recommendation system 506 to generate content recommendations 508. Evaluation application 220 computes, based on user feedback 514, one or more incremental impact metrics that indicate the incremental impact of including content 524 in content recommendations 508. In some embodiments, the incremental impact metric(s) include a content performance incremental lift metric that indicates an incremental performance lift from a particular content at a particular position, calculated as the difference between a Self-Normalized Inverse Propensity Score (SNIPS) for the particular content and the SNIPS for a random content at the same position. The SNIPS for a particular content is calculated as an averaged reward indicating user engagement with the particular content when the particular content is shown, weighted by the probability with which the particular content is shown according to a policy, and normalized. In some embodiments, the normalization can be omitted if, for example, stratified sampling is used. In some embodiments, the incremental impact metric(s) include a leave-one-out content performance lift metric that is calculated asNETF0373PCthe incremental SNIPS performance lift between a target policy for selecting content from a content set according to a probability and a leave-one-out (LOO) policy that simulates a counterfactual scenario where specific content is removed from the content set. In some embodiments, the incremental impact metric(s) include a leave-one-out content set performance lift metric that indicates a difference between a SNIPS content set performance of the full policy where recommendation system 506 has access to the full content set and the SNIPS performance of a leave-one-out policy where recommendation system 506 has access to all content except the content of interest.
[0044] More specifically, in some embodiments, evaluation application 220 computes incremental impact metric(s) in a manner that accounts for incrementality and bias reduction for recommended content, while being agnostic to model exposure. In such cases, evaluation application 220 can implement a bandit / reinforcement learning (RL) setup, where each arm atis selected following policy n with probability 7T(afI xf) given context xt. If atis chosen, reward rfwill be observed, which can be a short-term reward such as click-through rate (CTR) or a long-term reward such as user retention rate. For bias reduction, the Inverse Propensity Score (IPS), or a clipped variation of IPS, can be used to correct for over-or under-representations of content. For a target policy ne(that recommendation system 506 uses) under logged policy n0(that is the explore policy where content is selected to recommend with a level of randomness), a SNIPS can be computed as:yn e‘ai, n0)y\ ynm(ai,ne'ai,n0)SNIPS.i=1TTofciilXi) / i= 1 17t0(ai\Xi') ’ (Equation 1)where m is the match function that evaluates the degree of alignment between the target policy and logged policy. Using SNIPS, evaluation application 220 can compute one or more incremental impact metrics that assess the incremental impact of content 524.
[0045] In some embodiments, evaluation application 220 computes a content performance incremental lift metric. Assuming a content set c is reviewed by users, which includes N pieces of content shown in K positions, evaluation application 220 can consider the incremental performance lift from the content ctat position k as the difference between the associated SNIPS estimate and the SNIPS of a randomNETF0373PCcontent from the content set c at the same position to mitigate position bias and presentation bias. As the content may appear in different positions, evaluation application 220 can aggregate the lift across all possible positions for content ct, weighted by the wk(estimated by a debiasing technique) for each position k, giving the following content performance incremental lift metric:wfc(SNIPSc. fc- SNIPSc;k). (Equation 2)
[0046] In some embodiments, evaluation application 220 computes a leave-one-out content performance lift metric. The leave-one-out content performance lift metric calculates the SNIPS lift between a target policy and a LOO policy. The leave-one-out policy simulates a counterfactual scenario where specific content is removed from the content set. Specifically, for exact match, the match function will take a value of 1 if: (1) the explored arm is the best recommendation and the explored arm is not the removed content, or (2) the explored arm is the second best recommendation and the best recommendation is the removed content. Similar to the performance lift metric described above, the leave-one-out performance lift is first calculated at each position k, and then aggregated across all possible values of k :wfc(SNIPSC fc- SNIPSc.gC fc). (Equation 3)
[0047] In some embodiments, evaluation application 220 computes a leave-one-out content set performance lift metric. Rather than aggregating the individual leave-one-out performance lift, an alternative approach is to approximate the content recommendation process using explore data. The leave-one-out content set performance lift metric directly formulates the SNIPS content set performance of the full policy minus the policy where models 526 have access to the full content set and the SNIPS performance of the leave-one-out policy where models 526 have access to all content except the content of interest. The full policy can be approximated by selecting the best content for each k. Exact matching is a special case that provides unbiased estimates. However, other matching functions can be employed for various bias-variance tradeoffs. The leave-one-out policy can be approximated by selecting the best content, excluding the content of interest, for each k. The similarity S can then be calculated between the full policy and the explore policy, 5(p( ),p(e)), as well as the similarity between the leave-one-out policy and the explore policy,NETF0373PCS(p(Zoo),p(e)). The SNIPS lift of the full policy and the leave-one-out policy can be calculated accordingly.
[0048] Once computed, the incremental impact metric(s) can be used in any suitable manner. In some embodiments, the incremental impact metric(s) can be provided as feedback 528 to users 502. For example, evaluation application 220 could display the incremental impact metric(s) for different content candidates 504 created by users 502 to those users 502 via a user interface. In some embodiments, the incremental impact metric(s) can be used to improve recommendation system 506. For example, the incremental impact metric(s) could be used to debug the performance of trained machine learning models in recommendation system 506, such as to identify blind spots in which content that is impactful according to the incremental impact metric(s) is underrepresented in content recommendations generated using those machine learning models. In such cases, the machine learning models can be re-trained or otherwise updated based on the debugging information to improve performance of the machine learning models.
[0049] FIG. 6 sets forth a flow diagram of method steps for evaluating content recommendations, according to various embodiments. Although the method steps are described in conjunction with the systems of FIGS. 1-5, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.
[0050] As shown, a method 600 begins with step 602, where server application 217 generates explore content recommendations 510. In some embodiments, server application 217 can generate explore content recommendations 510 by determining, for each of a number of positions, a ranking of candidate content for the position, and selecting a content recommendation for the position based on the ranking and a probability distribution, as discussed in greater detail below in conjunction with Figure 7. In some embodiments, server application 217 can generate content recommendations 508 that include a certain percentage (e.g., ten percent) of explore content recommendations 510 and the rest being exploit content recommendations 512 that include deterministically selected content recommendations, such as content recommendations according to a ranking determined using one or more trained machine learning models.NETF0373PC
[0051] At step 604, server application 217 serves explore content recommendations 510 to playback applications 436. In some embodiments, server application 217 can also serve exploit content recommendations 512 to playback applications 436. In some embodiments, the recommendation system 506 can generate explore content recommendations 510 and exploit content recommendations 512 based on a configurable frequency. For example, a certain percentage of content recommendations 508 could be explore content recommendations 510, and the rest of content recommendations 508 could be exploit content recommendations 512.
[0052] At step 606, playback applications 436 present explore content recommendations 510 to users. Explore content recommendations 510 can be presented in any technically feasible manner in some embodiments, such as via user interfaces.
[0053] At step 608, playback applications 436 receive user feedback 514 and transmit user feedback 514 to evaluation application 220. In some embodiments, user feedback 514 includes user selections associated with content recommendations 508. In some embodiments, user feedback 514 can also include other information, such as dwell time, engagement rate, abandonment rate, user retention rate, and / or other signals.
[0054] At step 610, evaluation application 220 computes incremental impact metric(s) based on user feedback 514. In some embodiments, the incremental impact metric(s) include one or more of (1) a content performance incremental lift metric that indicates an incremental performance lift from a particular content at a particular position, calculated as the difference between a SNIPS for the particular content and the SNIPS for a random content at the same position; (2) a leave-one-out content performance lift metric that is calculated as the incremental SNIPS performance lift between a target policy for selecting content from a content set according to a probability and an LOO policy that simulates a counterfactual scenario where specific content is removed from the content set; or (3) a leave-one-out content set performance lift metric that indicates a difference between a SNIPS content set performance of a full policy where recommendation system 506 has access to the full content set and the SNIPS performance of a leave-one-out policy where recommendation system 506 has access to all content except a content of interest. InNETF0373PCsuch cases, the SNIPS for a particular content is calculated as a reward indicating user engagement with the particular content, weighted by the probability with which the particular content is shown according to a policy, and normalized. In some embodiments, the normalization can be omitted if, for example, stratified sampling is used. In some embodiments, the incremental impact metric(s) can be computed as described above in conjunction with Figure 5. Once computed, the incremental impact metric(s) can be used in any suitable manner, such as being provided as feedback to users who created content and / or used to debug machine learning models that generate content recommendations.
[0055] FIG. 7 sets forth a flow diagram of method steps for generating explore content recommendations, according to various embodiments. Although the method steps are described in conjunction with the systems of FIGS. 1-5, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.
[0056] As shown, step 602 of method 600 begins with step 702, where server application 217 determines a ranking of candidate content for a position. Any suitable content can be ranked, and the position can be a position on a page in some embodiments. For example, the content could include rows of media content items, and the position could be one of a number of positions on a page where rows can be placed. In some embodiments, recommendation system 506 of server application 217 can apply one or more trained machine learning models to determine the ranking of candidate content based on information associated with content being recommended and user information, such that content predicted to be of interest to particular users is recommended to those users.
[0057] At step 704, server application 217 selects a content recommendation for the position based on the ranking of content candidates and a probability distribution. In some embodiments, recommendation system 506 of server application 217 determines a ranking of candidate content for each position on a page.Recommendation system 506 selects, for the position, a candidate row that has not previously been used based on the determined ranking for the candidate rows and a probability distribution. In some embodiments, the probability distribution indicates the probability with which each rank is selected, and any suitable probability distribution, such as a Boltzmann distribution, can be used.NETF0373PC
[0058] At step 706, if there are more positions, then step 602 returns to step 702, where server application 217 determines a ranking of candidate content for another position.
[0059] In sum, techniques are disclosed for recommendation system observability. In some embodiments, a server application generates and provides, to playback applications, content recommendations that include (1) explore content recommendations used to determine the incremental impact of content being recommended, and (2) exploit content recommendations of content that is most relevant to users. The playback applications present the content recommendations to users and transmit user feedback, such as user selections, associated with the explore content recommendations to an evaluation application. The evaluation application computes, based on the user feedback, one or more incremental impact metrics that indicate the incremental impact of content in content recommendations. After computation, the incremental impact metric(s) can be used in any suitable manner, such as by being provided as feedback to users who generated content and / or being used to debug machine learning models that generate content recommendations.
[0060] At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable the performance of content that is recommended by recommendation systems to be measured. The measured performance of content can be used to debug, and then to re-train or otherwise update, machine learning models used by the recommendation systems to generate improvement recommendations. The measured performance of content can also be output to improve user understanding of the effectiveness of content that was recommended. These technical advantages provide one or more technological improvements over prior art approaches.
[0061] 1. In some embodiments, a computer-implemented method for recommendation system observability comprises generating, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations, transmitting the one or more content recommendations to one or more endpoint devices, receiving, from the one or more endpoint devices, feedback associated with user engagement with a first content recommended in the one or more content recommendations, and computing, based at least on theNETF0373PCfeedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
[0062] 2. The computer-implemented method of clause 1 , wherein the metric is based at least on one or more computations of a score based at least on an averaged reward associated with user engagement with given content, wherein the score is weighted by a probability with which the given content is recommended, and wherein the score is normalized.
[0063] 3. The computer-implemented method of clauses 1 or 2, wherein the metric is based at least on a difference between a first computation of the score for the first content and a second computation of the score for a second content at a same position as the first content.
[0064] 4. The computer-implemented method of any of clauses 1-3, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with one content removed.
[0065] 5. The computer-implemented method of any of clauses 1-4, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with the first content removed.
[0066] 6. The computer-implemented method of any of clauses 1-5, further comprising generating, based at least on another output of the recommendation system, one or more other content recommendations, and transmitting the one or more other content recommendations to one or more other endpoint devices.
[0067] 7. The computer-implemented method of any of clauses 1-6, wherein the feedback includes at least one of user selections, dwell time, engagement rate, abandonment rate, or retention rate associated with the first content.
[0068] 8. The computer-implemented method of any of clauses 1-7, wherein the first content includes a row of media content items.NETF0373PC
[0069] 9. The computer-implemented method of any of clauses 1-8, further comprising performing, based at least on the metric, one or more operations to train one or more machine learning models included in the recommendation system to generate one or more trained machine learning models.
[0070] 10. The computer-implemented method of any of clauses 1-9, further comprising outputting the metric via a display device.
[0071] 11. In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of generating, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations, transmitting the one or more content recommendations to one or more endpoint devices, receiving, from the one or more endpoint devices, feedback associated with user engagement with a first content recommended in the one or more content recommendations, and computing, based at least on the feedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
[0072] 12. The one or more non-transitory computer-readable media of clause 11 , wherein the metric is based at least on one or more computations of a score based at least on an averaged reward associated with user engagement with given content, wherein the score is weighted by a probability with which the given content is recommended, and wherein the score is normalized.
[0073] 13. The one or more non-transitory computer-readable media of clauses 11 or 12, wherein the metric is based at least on a difference between a first computation of the score for the first content and a second computation of the score for a second content at a same position as the first content.
[0074] 14. The one or more non-transitory computer-readable media of any of clauses 11-13, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with one content removed.NETF0373PC
[0075] 15. The one or more non-transitory computer-readable media of any of clauses 11-14, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with the first content removed.
[0076] 16. The one or more non-transitory computer-readable media of any of clauses 11-15, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the steps of generating, based at least on another output of the recommendation system, one or more other content recommendations, and transmitting the one or more other content recommendations to one or more other endpoint devices.
[0077] 17. The one or more non-transitory computer-readable media of any of clauses 11-16, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of performing, based at least on the metric, one or more operations to train one or more machine learning models included in the recommendation system to generate one or more trained machine learning models.
[0078] 18. The one or more non-transitory computer-readable media of any of clauses 11-17, wherein the one or more content recommendations include a first percentage of overall content recommendations, and a second percentage of the overall content recommendations is generated deterministically.
[0079] 19. The one or more non-transitory computer-readable media of any of clauses 11-18, wherein the output of the recommendation system includes a ranking of content for a position.
[0080] 20. In some embodiments, a system comprises one or more memories storing instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions generate, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations, transmit the one or more content recommendations to one or more endpoint devices, receive, from the one or more endpoint devices, feedbackNETF0373PCassociated with user engagement with a first content recommended in the one or more content recommendations, and compute, based at least on the feedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
[0081] Any and all combinations of any of the claim elements recited in any of the claims and / or any elements described in this application, in any fashion, fall within the contemplated scope of the present disclosure and protection.
[0082] The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.
[0083] Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module,” a “system,” or a “computer.” In addition, any hardware and / or software technique, process, function, component, engine, module, or system described in the present disclosure may be implemented as a circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
[0084] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-onlyNETF0373PCmemory (EPROM or Flash memory), an optical fiber, a portable compact disc readonly memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
[0085] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / acts specified in the flowchart and / or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.
[0086] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may include a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specifiedNETF0373PCfunctions or acts, or combinations of special purpose hardware and computer instructions.
[0087] While the preceding is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Claims
NETF0373PCWHAT IS CLAIMED IS:
1. A computer-implemented method for recommendation system observability, the method comprising:generating, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations; transmitting the one or more content recommendations to one or more endpoint devices;receiving, from the one or more endpoint devices, feedback associated with user engagement with a first content recommended in the one or more content recommendations; andcomputing, based at least on the feedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
2. The computer-implemented method of claim 1 , wherein the metric is based at least on one or more computations of a score based at least on an averaged reward associated with user engagement with given content, wherein the score is weighted by a probability with which the given content is recommended, and wherein the score is normalized.
3. The computer-implemented method of claim 2, wherein the metric is based at least on a difference between a first computation of the score for the first content and a second computation of the score for a second content at a same position as the first content.
4. The computer-implemented method of claim 2, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with one content removed.
5. The computer-implemented method of claim 2, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computationNETF0373PCof the score for a leave-one-out policy in which the recommendation system has access to the content set with the first content removed.
6. The computer-implemented method of claim 1 , further comprising:generating, based at least on another output of the recommendation system, one or more other content recommendations; andtransmitting the one or more other content recommendations to one or more other endpoint devices.
7. The computer-implemented method of claim 1 , wherein the feedback includes at least one of user selections, dwell time, engagement rate, abandonment rate, or retention rate associated with the first content.
8. The computer-implemented method of claim 1 , wherein the first content includes a row of media content items.
9. The computer-implemented method of claim 1 , further comprising performing, based at least on the metric, one or more operations to train one or more machine learning models included in the recommendation system to generate one or more trained machine learning models.
10. The computer-implemented method of claim 1 , further comprising outputting the metric via a display device.
11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:generating, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations; transmitting the one or more content recommendations to one or more endpoint devices;receiving, from the one or more endpoint devices, feedback associated with user engagement with a first content recommended in the one or more content recommendations; andNETF0373PCcomputing, based at least on the feedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
12. The one or more non-transitory computer-readable media of claim 11 , wherein the metric is based at least on one or more computations of a score based at least on an averaged reward associated with user engagement with given content, wherein the score is weighted by a probability with which the given content is recommended, and wherein the score is normalized.
13. The one or more non-transitory computer-readable media of claim 12, wherein the metric is based at least on a difference between a first computation of the score for the first content and a second computation of the score for a second content at a same position as the first content.
14. The one or more non-transitory computer-readable media of claim 12, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with one content removed.
15. The one or more non-transitory computer-readable media of claim 12, wherein the metric is based at least on a difference between a first computation of the score for a first policy in which the recommendation system has access to a content set and a second computation of the score for a leave-one-out policy in which the recommendation system has access to the content set with the first content removed.
16. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the steps of:generating, based at least on another output of the recommendation system, one or more other content recommendations; andtransmitting the one or more other content recommendations to one or more other endpoint devices.NETF0373PC17. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of performing, based at least on the metric, one or more operations to train one or more machine learning models included in the recommendation system to generate one or more trained machine learning models.
18. The one or more non-transitory computer-readable media of claim 11 , wherein the one or more content recommendations include a first percentage of overall content recommendations, and a second percentage of the overall content recommendations is generated deterministically.
19. The one or more non-transitory computer-readable media of claim 11 , wherein the output of the recommendation system includes a ranking of content for a position.
20. A system, comprising:one or more memories storing instructions; andone or more processors that are coupled to the one or more memories and, when executing the instructions:generate, based at least on an output of a recommendation system and a probability distribution, one or more content recommendations, transmit the one or more content recommendations to one or more endpoint devices,receive, from the one or more endpoint devices, feedback associated with user engagement with a first content recommended in the one or more content recommendations, andcompute, based at least on the feedback, a metric indicating an incremental impact of the first content in the one or more content recommendations.
