Fixed image conversion

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for converting fixed images. An initial image having a set aspect ratio is obtained. The set aspect ratio of the initial image differs from a target aspect ratio of a target presentation area. A still image of a video is obtained. The initial image is masked with masking pixels indicating where additional image content is required to be generated. A reformatted image having the second aspect ratio of the target presentation area is generated based on application of a generative model to the masked image, including generating new pixels for the reformatted image based on content of the initial image and content of the still image. The reformatted image is inserted into the target presentation area resulting in a visual presentation that includes both of the reformatted image and the video.

Description

Attorney Docket No. t>6113-087t>WOlFIXED IMAGE CONVERSIONBACKGROUND

[0001] The present disclosure relates to data processing, and more specifically, modifying characteristics of fixed images. Images are displayed online, and can be displayed with content provided by different parties. For example, a web page can incorporate images from various sources and display them with native content of the web page. The same is true in the native application environment.SUMMARY

[0002] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of obtaining, by a system including electrical circuitry, an initial image having a set aspect ratio; determining, by the system, that the set aspect ratio of the initial image differs from a target aspect ratio of a target presentation area; obtaining, by the system, a still image of a video with which the target presentation area is presented; masking, by the system, the initial image with masking pixels indicating where additional image content is required to be generated, wherein the masking results in a masked image that includes pixels of the initial image and the masking pixels; generating, by the system, a reformatted image having the second aspect ratio of the target presentation area based on application of a generative model to the masked image, including generating new pixels for the reformatted image based on content of the initial image and content of the still image; and inserting, by the system, the reformatted image into the target presentation area resulting in a visual presentation that includes both of the reformatted image and the video.

[0003] These and other embodiments can each optionally include one or more of the following features. Methods can include the operations of training the generative model based on a set of training images. The training can include obtaining a reference image; masking the reference image, wherein the masking of the reference image creates a training image; applying the generative model to the training image, wherein application of the generative model to the training image creates a corresponding test image; and determining an error value representing differences between the reference image and the correspondingAttorney Docket No. t>6113-087t>WOltest image created using the training image. Methods can further include adjusting parameters of the generative model based on the error value.

[0004] Masking the reference image can include masking a first set of pixels of the reference image indicating where additional image content is required to be generated.Determining the error value representing differences between the training image and the corresponding test image created using the training image can include determining the error value based on differences between (i) the additional content generated at locations of the first set of pixels and (ii) reference content of the reference image.

[0005] Masking the reference image can include padding the reference image with a border ratio in an outpainting training cycle; and masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle.

[0006] Masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle can include masking the reference image with at least one of (i) two random non-overlapping bounding boxes or (ii) one horizontally rectangular bounding box.

[0007] Methods can further include the operations of determining a reference timestamp of the video at which the still image of the video appears; storing the reformatted image with a reference to the reference timestamp; receiving a request for an image, wherein the request identifies the video and includes a current playback timestamp; and determining a time difference between the current playback timestamp and the reference timestamp. Inserting the reformatted image into the target presentation area can be performed based on the time difference being less than a specified threshold.

[0008] Methods can further include rearranging visual features of the initial image based on a location of a frame within which the video is visually presented during a pause of the video.

[0009] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.Attorney Docket No. t>6113-087t>WOlBRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. l is a block diagram of an example environment in which image conversion can be performed.

[0011] FIG. 2 is a block diagram illustrating the creation of training pairs of images and training a generative model.

[0012] FIG. 3 is an illustration of image masking and image reformatting using outpainting.

[0013] FIG. 4 is an illustration of image masking and image reformatting using inpainting.

[0014] FIG. 5A is an illustration of an example video streaming site user interface for which the image reformatting can be used to create digital components for presentation.

[0015] FIG. 5B is another illustration of an example video streaming site user interface for which the image reformatting can be used to create digital components for presentation.

[0016] FIG. 6 is a flow chart of an example process for reformatting images.

[0017] FIG. 7 is a flow chart of an example process for reformatting images.

[0018] FIG. 8 is a block diagram of an example computer system.

[0019] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0020] FIG. l is a block diagram of an example environment in which image conversion can be performed. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, user devices 106, digital component servers 108, and a service system 110. The example environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0021] A client device 106 is an electronic device that is capable of requesting and receiving online resources over the network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over the network 102. A client device 106 typically includes a userAttorney Docket No. t>6113-087t>WOlapplication, such as a web browser, to facilitate the sending and receiving of data over the network 102, but native applications (other than browsers) executed by the client device 106 can also facilitate the sending and receiving of data over the network 102.

[0022] A gaming device is a device that enables a user to engage in gaming applications, for example, in which the user has control over one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (either physical or visually rendered) that enables user control over content rendered by the gaming application. The gaming device can store and execute the gaming application locally, or execute a gaming application that is at least partly stored and / or served by a cloud server (e.g., online gaming applications). Similarly, the gaming device can interface with a gaming server that executes the gaming application and “streams” the gaming application to the gaming device. The gaming device may be a tablet device, mobile telecommunications device, a computer, or another device that performs other functions beyond executing the gaming application.

[0023] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input by way of voice, and respond with content using audible feedback, and can present other audible information. In some situations, digital assistant devices also include a visual display or are in communication with a visual display (e g., by way of a wireless or wired connection). Feedback or other information can also be provided visually when a visual display is present. In some situations, digital assistant devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices that are registered with the digital assistant device.

[0024] As illustrated, the client device 106 is presenting an electronic document 150. An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., “apps” and / or gaming applications), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided to client devices 106 by electronic document servers 104 (“Electronic Doc Servers”).Attorney Docket No. t>6113-087t>WOl

[0025] For example, the electronic document servers 104 can include servers that host publisher websites. In this example, the client device 106 can initiate a request for a given publisher webpage, and the electronic server 104 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device 106.

[0026] In another example, the electronic document servers 104 can include app servers from which client devices 106 can download apps. In this example, the client device 106 can download fdes required to install an app at the client device 106, and then execute the downloaded app locally. Alternatively, or additionally, the client device 106 can initiate a request to execute the app, which is transmitted to a cloud server. In response to receiving the request, the cloud server can execute the application and stream a user interface of the application to the client device 106 so that the client device 106 does not have to execute the app itself. Rather, the client device 106 can present the user interface generated by the cloud server’s execution of the app, and communicate any user interactions with the user interface back to the cloud server for processing.

[0027] Electronic documents can include a variety of content. For example, an electronic document 150 can include native content 152 that is within the electronic document 150 itself and / or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a script, such as the script 154, that causes the client device 106 to request content (e.g., digital component) from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., digital component) obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

[0028] In some situations, a given electronic document (e.g., electronic document 150) can include a digital component script (e.g., script 154) that references the service system 110, or a particular service provided by the service system 110. In these situations, the digital component script is executed by the client device 106 when the given electronicAttorney Docket No. t>6113-087t>WOldocument is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for digital components 112 (referred to as a “component request”), which is transmitted over the network 102 to the service system 110. For example, the digital component script can enable the client device 106 to generate a packetized data request including a header and payload data. The component request 112 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a device identifier corresponding the computing device 106, such as a name (or network location, such as an Internet Protocol address) of the requesting device (e.g., the computing device 106), a cookie of the service system 110 that was previously placed on the computing device by the service system 110, and / or information that the service system 110 can use to select one or more digital components, or other content, provided in response to the request. In some implementations, the cookie that is included in the component request 112 can be a cookie corresponding to an account of a user that is logged into the service system 110, which is referred to as a service cookie. The component request 112 is transmitted, by the client device 106, over the network 102 (e.g., a telecommunications network) to a server of the service system 110.

[0029] The component request 112 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and / or media types that are eligible for presentation in the locations can be provided to the service system 110. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 112 (e.g., as payload data) and provided to the service system 110 to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from the client device 106 to obtain a search results page.Attorney Docket No. t>6113-087t>WOl

[0030] Component requests 112 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests 112 can be transmitted, for example, over a packetized network, and the component requests 112 themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the payload data can include any of the information discussed above.

[0031] The service system 110 chooses digital components (e g., third-party content, such as video fdes, audio fdes, images, text, and combinations thereof, which can all take the form of advertising content or non-advertising content) that will be presented with the given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and / or using information included in the component request 112.

[0032] In some implementations, a digital component is selected in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request 112 can result in page load errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 106.

[0033] Also, as the delay in providing the digital component to the client device 106 increases, it is more likely that the electronic document will no longer be presented at the client device 106 when the digital component is delivered to the client device 106, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device 106 when the digital component is provided.

[0034] In some implementations, the service system 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computingAttorney Docket No. t>6113-087t>WOldevices 114 that are interconnected and identify and distribute digital component in response to requests 112. The set of multiple computing devices 114 operate together to identify a set of digital components that are eligible to be presented in the electronic document from among a corpus of millions of available digital components (DCl-x). The millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry can reference the corresponding digital component and / or include distribution parameters (DPl-DPx) that contribute to (e.g., trigger, condition, or limit) the distribution / transmission of the corresponding digital component. For example, the distribution parameters can contribute to (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

[0035] In some implementations, the distribution parameters for a particular digital component can include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 112) in order for the digital component to be eligible for presentation. Additionally, or alternatively, the distribution parameters can include embeddings that can use various different dimensions of data, such as website details and / or consumption details (e.g., page viewport, user scrolling speed, or other information about the consumption of data). The distribution parameters can also require that the component request 112 include information specifying a particular geographic region (e.g., country or state) and / or information specifying that the component request 112 originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation. The distribution parameters can also specify an eligibility value (e.g., ranking score, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution / transmission (e.g., among other available digital components).

[0036] The identification of the eligible digital component can be segmented into multiple tasks 117a-l 17c that are then assigned among computing devices within the set of multiple computing devices 114. For example, different computing devices in the set 114 can each analyze a different portion of the digital component database 116 to identify various digital components having distribution parameters that match information included in theAttorney Docket No. t>6113-087t>WOlcomponent request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1-Res 3) 118a- 118c of the analysis back to the service system 110. For example, the results 118a-l 18c provided by each of the computing devices in the set 114 may identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of the digital component that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data.

[0037] The service system 110 aggregates the results 118a- 118c received from the set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more digital components that will be provided in response to the request 112. For example, the service system 110 can select a set of winning digital components (one or more digital components) based on the outcome of one or more content evaluation processes, as discussed below. In turn, the service system 110 can generate and transmit, over the network 102, reply data 120 (e.g., digital data representing a reply) that enable the client device 106 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components (e.g., winning third-party content) and the content of the electronic document are presented together at a display of the client device 106.

[0038] In some implementations, the client device 106 executes instructions included in the reply data 120, which configures and enables the client device 106 to obtain the set of winning digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 can include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing multiple digital components) and transmit,Attorney Docket No. t>6113-087t>WOlto the client device 106, digital component data (DC Data) 122 that presents the given winning digital component in the electronic document at the client device 106.

[0039] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content), and present the digital component at a location specified by, or assigned to, the script 154. For example, the script 154 can create a walled garden environment, such as a frame, that is presented within, e.g., besides the native content 152 of the electronic document 150. In some implementations, the digital component is overlayed over (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service system 110 can specify the presentation location within the electronic document 150 in the reply 120.

[0040] It is desirable for the digital component to be formatted in a way that is visually complementary to the native content 152. To provide a digital component that is visually complementary to the native content 152, it may be necessary to reformat or otherwise modify various visual characteristics of the digital component, such as the aspect ratio, content of the digital component, and / or arrangement of objects / text presented by the digital component. However, digital components are often uploaded to the service system 110 in a finalized image format, such that the digital component is a “set-pixel image.” As used herein, a set-pixel image is an image made up of a fixed grid of pixels, where the aspect ratio (width-to-height proportion) and visual content are locked in place. As such, changing the proportions distorts the image, and the content cannot be edited directly without altering the original pixel data using external tools. When the digital component is uploaded as a setpixel image, also referred to as a fixed image or image with fixed characteristics, it can be technically challenging to reformat, or otherwise adjust the visual characteristics of the digital component based on the presentation constraints and / or native content 152.

[0041] Reformatting images with fixed characteristics (e.g., resolution, aspect ratio, color depth, or file format) presents several challenges because altering an image’s structure while preserving its core qualities (e.g., visual content and image quality) requires balancing technical constraints and visual fidelity. For instance, when resizing or changing the resolution (e.g., scaling a low-res image up or a high-res image down), it is possible that the image will lose sharpness or visual artifacts (like pixelation, blurriness, or jagged edges) willAttorney Docket No. t>6113-087t>WOlbe introduced into the image. Also, the fixed resolution of the image can limit how much the image can be stretched or compressed before the visual quality of the image degrades.

[0042] Similarly, if the original aspect ratio (width-to-height proportion) is fixed but the target format (e.g., presentation location specified by script 154) requires a different aspect ratio, the image much either be cropped, which results in lost content, or stretched, resulting in distorted content. For example, reformatting a 4:3 image into a 16:9 frame without cropping can make objects look unnaturally wide or tall. Meanwhile, cropping the image can result in a loss of important content (e.g., content that conveys meaningful information). Similarly, reformatting a detailed infographic into a smaller, fixed-resolution thumbnail could render text illegible or obscure key elements of the image.

[0043] To overcome these challenges, the system 100 can include an image conversion apparatus (“ICA”) 175. The ICA 175 includes hardware circuitry, such as computer processors, memory devices, network interfaces, and other circuitry that enables the ICA 175 to execute instructions and perform operations required to convert a fixed image into a reformatted image. In some implementations, the ICA 175 can include one or more generative models, such as one or more convolutional neural networks (CNNs) or diffusion models).

[0044] In some implementations, the ICA 175 can be configured to autonomously generate digital components, either prior to a request 112 (e.g., offline) and / or in response to a request 112 (e.g., online or real-time). As described in more detail throughout this specification, the ICA 175 can generate the digital component based on an existing digital component. For example, the one or more generative models of the ICA 175 can use an existing fixed image (e.g., initial image) and information about a target presentation area to generate a reformatted image that will be delivered to a client device for presentation in the target presentation area. As part of the transformation, the one or more models can be trained to perform inpainting and outpainting.

[0045] Inpainting refers to the process of filling in missing or damaged parts of an image to create a seamless, coherent result. Models like Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), or more recent diffusion models (e.g., Stable Diffusion) can be used to perform inpainting. These models are generally trained on vast datasets of images and learn to predict and generate plausible content to replace missing regions basedAttorney Docket No. t>6113-087t>WOlon surrounding context. However, absent a large data set, the effectiveness of these models in performing inpainting tasks can be limited.

[0046] Outpainting refers to extending an image beyond its original boundaries by generating new content that blends naturally with the existing / original content of the image. Like inpainting, image-based generative models, such as diffusion models, GANs, and transformer-based vision models can be used to perform outpainting. These models analyze the edges and context of the initial / existing image to extrapolate and create additional content, such as extending a landscape or adding to a scene.

[0047] In the context of converting digital components, there is sometimes a lack of image training pairs, referred to as ground truth pairs, on which the models can be trained to perform inpainting and outpainting. Given this lack of training pairs, it is difficult to train models to perform inpainting and outpainting in the context of digital components. To overcome this challenge, the ICA 175 can use masking techniques to create training pairs / ground truth pairs, which can include the original image / reference image and a masked image / training image, as discussed in more detail below. Those training pairs can then be used to train the models. For instance, the model can take the masked image as input and generate content for the masked portion of the masked image, and output a test image. An error value can be generated for the test image based on visual differences between the test image and the reference image, and the parameters of the model can be adjusted based on the error value. The generation of test images and adjustment of model parameters can continue until a stop condition is met, thereby indicating that the model is sufficiently trained. The stop condition can be, for example, reaching a target error value, completing a specified number of iterations, or both.

[0048] Once the model is sufficiently trained, the model can be used to generate reformatted images that can be delivered to client devices 106 in response to requests 112. For example, the reformatted images can be generated to fit within a specified presentation area of the electronic document 150. The reformatted images can also be generated to have a color scheme that is similar to a color scheme of the native content 152 of the electronic document 150, thereby resulting in a more uniform color scheme for the combined presentation of the native content 152 and the digital component. As discussed in more detail below, the reformatting of the image can include one or more of adjusting an aspect ratio ofAttorney Docket No. t>6113-087t>WOlthe image, adjusting a layout of objects presented in the image, and / or modifying objects presented in the image. The modifications can be made, for example, based on factors such as whether portions of the image will be occluded by the native content 152, areas of the electronic document 150 that are available for presentation of the reformatted image (e.g., digital component), and display characteristics of the native content 152 that will be visually presented / visually perceivable together with the reformatted image.

[0049] FIG. 2 is a block diagram 200 illustrating the creation of training pairs of images and training a generative model. In some implementations, the training pairs of images are created by the image conversion apparatus 175. Each training pair of images includes a reference image and a training image. The reference image can be, for example, an original version of the image, and the training image can be a masked version of the training image. The discussion that follows discusses the creation of an inpainting training pair that is used to train models to perform inpainting, as well as the creation of an outpainting training pair that is used to train models to perform outpainting. The discussion that follows will refer to the same reference image 202 for both of the inpainting example and the outpainting example. Different reference images can also be used to create different training image pairs.

[0050] The image apparatus 175 obtains the reference image 202 from a reference image database 204. The reference image database 204 includes a hardware memory device that stores a set of reference images. The set of reference images can be selected based on visual quality and / or other characteristics. For example, the set of reference image can be a set of images that have been classified as images having high visual quality scores, such as scores indicating that the image has a specified level of clarity, brightness, contrast, and / or other visual qualities. The set of reference images can also be identified, at least in part, based on historical user interactions with the images. For example, each of the images can be scored based on a level of user interaction with the image (e.g., number of users clicks of the image) when presented as a digital component. The images classified as reference images can be required to have at least a specified level of user interactions.

[0051] The image conversion apparatus 175 can be configured (e.g., specially programmed) to mask the reference image 202 to create training images 206 and 208. The training image 206 is an example of an outpainting training image, and the training image 208 is an example of an inpainting training image. The training image 206 can be used toAttorney Docket No. t>6113-087t>WOltrain a model to perform outpainting, while the training image 208 can be used to train a model to perform inpainting.

[0052] In the outpainting scenario, the image conversion apparatus 175 creates the training image 206 by applying masking borders of the reference image 202. For example, as shown, the image conversion apparatus 175 can apply the mask 210 to the top edge of the reference image 202 and apply the mask 212 to the bottom edge of the reference image 202. The application of the masks 210 and 212 can be referred to as padding the reference image 202 with a border ratio. The masked version of the reference image 202 is the training image 206, which has masking occluding a top segment of the reference image 202 and a bottom segment of the reference image 202. The image conversion apparatus 175 can pair the reference image 202 with the training image 206 to create an outpainting training pair that can be used to train a model to perform image outpainting, and store the training pair in a training pairs database 214.

[0053] The image conversion apparatus 175 can input the training image 206 into a generative model 216 with model instructions that cause the generative model to create an output image having dimensions of the reference image, or with instructions to create pixels in the areas of the masks 210 and 212. The generative model 216 can be included in the image conversion apparatus 175, or be implemented in a different apparatus.

[0054] The generative model 216 will use the training image 206 and the instructions to generate pixels to fdl the areas of the masks 210 and 212, thereby creating, as an output, a test image 218 having the aspect ratio of the reference image 202. The test image 218 can be an image that is fully generated by the generative model 216, or an image that combines the portion of the reference image 202 that was included in the training image (e.g., the portion of the reference image that was not masked) and pixels generated by the generative model 216 for the areas of the masks 210 and 212. In some situations, combining the pixels generated for the areas of the masks 210 and 212 with the portion of the reference image 202 that was not at the locations of the masks 210 and 212 enables the evaluation of the generated content to be focused on only those portions of the test image that needed to be generated, rather than enabling errors to be introduced in portions of the image that did not need to be generated by the generative model 216.Attorney Docket No. t>6113-087t>WOl

[0055] The image conversion apparatus 175 can determine an error value based on differences between the test image 218 and the reference image 202. For example, the image conversion apparatus 175 can perform a pixel -wise loss calculation, such as a mean squared error or mean absolute error calculation that directly compares generated pixel values of the test image 218 in the areas of the masks 210 and 212 with pixels of the reference image 202 that are in the same locations of the generated pixels of the test image 218. The image conversion apparatus 175 can also perform a perceptual loss determination, which compares higher level features (e.g., relative to a pixel-wise comparison) of features such as texture, edges, etc.) that are extracted by a pretrained model, rather than (or in addition to) performing the pixel-wise loss calculation. The image conversion apparatus 175 can also perform an adversarial loss determination, which is an assessment of how “real” the generated portion of the test image 218 appears relative to the same portions of the reference image. The error value can represent the differences between the reference image and the corresponding test image. In some implementations, the error value is a sum of the differences determined using one or more of the loss calculations discussed above.

[0056] The image conversion apparatus 175 uses the error value to adjust the parameters of the generative model 216. In some implementations, the image conversion apparatus 175 adjusts the model parameters in the direction that reduces the loss (e.g., the error value). Generally speaking, this can be performed by computing gradients that indicate how much each model parameter contributed to the error, and subtracting a small portion of the gradient from the current weights of the model parameters. The image conversion apparatus 175 can iterate the generation of test images and adjustment of model parameters until a stop condition is reached. At that point, the image conversion apparatus 175 can deploy the generative model 216 to reformat images.

[0057] In the inpainting scenario, the image conversion apparatus 175 creates the training image 208 by applying masking to interior portions of the reference image 202. That is, instead of applying masking to the edge pixels and some specified amount of pixels around the edges of the reference image 202, the image conversion apparatus 175 will insert the masks 218 and 220 at locations that are at least a specified distance (e.g., number of pixels) from the edges of the reference image 202 to create the training image 108. For example, as shown, the image conversion apparatus 175 can apply the masks 220 and 222 to interiorAttorney Docket No. t>6113-087t>WOlportions of the reference image 202. In this scenario, the masked version of the reference image 202 is the training image 208, which has masking occluding two rectangular portions of the reference image 202. The image conversion apparatus 175 can pair the reference image 202 with the training image 208 to create an inpainting training pair that can be used to train a model to perform image inpainting, and store the inpainting training pair in a training pairs database 214.

[0058] The image conversion apparatus 175 can input the training image 208 into the generative model 216 with model instructions that cause the generative model to create an output image having dimensions of the reference image, or with instructions to create pixels in the areas of the masks 220 and 222. The generative model 216 can be included in the image conversion apparatus 175, or be implemented in a different apparatus. The generative model 216 in this case can be the same generative model that performed the outpainting, or can be a different generative model that is separately trained to perform inpainting.

[0059] The generative model 216 will use the training image 208 and the instructions to generate pixels to fdl the areas of the masks 220 and 222, thereby creating, as an output, a test image 218 having the aspect ratio of the reference image 202, and that fills in the missing content of the reference image 202 that was occluded (or removed) by the masks 220 and 222. The test image 218 can be an image that is fully generated by the generative model 216, or an image that combines the portion of the reference image 202 that was included in the training image 208 (e.g., the portion of the reference image that was not masked) and pixels generated by the generative model 216 for the areas of the masks 220 and 222. In some situations, combining the pixels generated for the areas of the masks 220 and 222 with the portion of the reference image 202 that was not at the locations of the masks 220 and 222 enables the evaluation of the generated content to be focused on those portions of the test image that needed to be generated, rather than enabling errors to be introduced in portions of the image that did not need to be generated by the generative model 216.

[0060] The image conversion apparatus 175 can again determine an error value based on differences between the test image 218 and the reference image 202. For example, the image conversion apparatus 175 can perform a pixel-wise loss calculation, such as a mean squared error or mean absolute error calculation that directly compares generated pixel values of the test image 218 in the areas of the masks 220 and 222 with pixels of the reference image 202Attorney Docket No. t>6113-087t>WOlthat are in the same locations of the generated pixels of the test image 218. The image conversion apparatus 175 can also perform a perceptual loss determination, which compares higher level features (e.g., relative to a pixel-wise comparison) of features such as texture, edges, etc.) that are extracted by a pretrained model, rather than (or in addition to) performing the pixel-wise loss calculation. The image conversion apparatus 175 can also perform an adversarial loss determination, which is an assessment of how “real” the generated portion of the test image 218 appears relative to the same portions of the reference image. The error value can represent the differences between the reference image and the corresponding test image. In some implementations, the error value is a sum of the differences determined using one or more of the loss calculations discussed above.

[0061] The image conversion apparatus 175 uses the error value to adjust the parameters of the generative model 216. In some implementations, the image conversion apparatus 175 adjusts the model parameters in the direction that reduces the loss (e.g., the error value). Generally speaking, this can be performed by computing gradients that indicate how much each model parameter contributed to the error, and subtracting a small portion of the gradient from the current weights of the model parameters. The image conversion apparatus 175 can iterate the generation of test images and adjustment of model parameters until a stop condition is reached. At that point, the image conversion apparatus 175 can deploy the generative model 216 to reformat images.

[0062] FIG. 3 is an illustration 300 of image masking and image reformatting using outpainting. The image conversion apparatus 175 is shown as performing the masking and reformatting. As previously discussed, the image conversion apparatus 175 can include a generative model that enables the reformatting, or the image conversion apparatus 175 can interact with one or more separate generative models to achieve the reformatting. For example, the image conversion apparatus 175 can be configured to perform all of the operations required to carry out the masking and reformatting, or the image conversion apparatus 175 can be configured to perform a portion of the operations (e.g., masking), while using a separate generative model to perform the reformatting.

[0063] As depicted by the illustration 300, the masking and reformatting are performed using an initial image 302. In some implementations, the initial image 302 is obtained by the image conversion apparatus 175. For example, the initial image 302 can be a fully renderedAttorney Docket No. t>6113-087t>WOldigital component in the form of a fixed image having one or more characteristics that are not directly editable without altering the original pixel data. For example, the initial image 302 can be an image having a given aspect ratio, such as 4:3, or another aspect ratio. The digital component may have been fully constructed and saved in an image format by a third party before being uploaded, or otherwise made accessible to the image conversion apparatus 175.

[0064] The image conversion apparatus 175 can determine that the initial image 302 needs to be reformatted. In some implementations, the image conversion apparatus 175 can determine that the initial image 302 needs to be reformatted based on one or more visual characteristics of the initial image 302 being non-compliant with one or more presentation requirements. For example, the image conversion apparatus 175 can determine that the current aspect ratio (e.g., set aspect ratio) of the initial image 302 differs from a target aspect ratio of a presentation area the image conversion apparatus 175 is populating. In this example, the current (set) aspect ratio of the initial image 302 can be 4:4 (e.g., a square), while the target aspect ratio of the presentation area may be 9: 16 (e.g., a vertical rectangle). In order for the image conversion apparatus 175 to use the initial image 302 to fully occupy the presentation area, the image conversion apparatus 175 must reformat the initial image 302. More specifically, the image conversion apparatus 175 needs to perform outpainting to create additional pixels at the top and bottom of the initial image 302, thereby extending the content of the initial image 302 in the vertical direction.

[0065] The image conversion apparatus 175 can apply masking to the initial image 302 to indicate where additional pixels need to be generated. As shown, the image conversion apparatus 175 can apply masks 304 and 306 to the top and bottom of the initial image 302 indicating that pixels need to be generated to extend content of the initial image 302 into the areas occupied by the masks 304 and 306. In some implementations, the image conversion apparatus 175 can also add masking 308 to the sides of the initial image 302. This masking 308 can be added to the sides of the initial image 302, for example, to maintain a same or similar aspect ratio as the initial image 302 and / or to ensure that the aspect ratio of the image being input to the generative model complies with aspect ratio requirements of the generative model. For example, if a generative model can only accept image inputs of a specified aspect ratio, this additional masking 308 can be used to ensure that the input image has the specified aspect ratio. In some situations, the additional masking can also be used to reduce distortionsAttorney Docket No. t>6113-087t>WOlthat may be generated by the generative model when reformatting from a square image to a rectangular image is performed.

[0066] The image conversion apparatus 175 can create a context mask 310 based on the masked version of the initial image 302. The masked version of the initial image 302 can also be referred to as a masked image. The context mask 310 includes a set of pixels indicating where content is to be generated, and a set of pixels indicating areas to ignore or exclude from the generative process. For example, white pixels of the mask areas 312 and 314 can indicate to the generative model that pixels should be generated to fill these areas. Meanwhile, the black pixels in the mask area 316 can indicate to the generative model that these portions of the input image are to be ignored or excluded from the image generation process. Recall that the mask pixels 308 were added to the sides of the initial image 302 (e.g., outside the border of the initial image 302), and therefore, no content exists in those area and because black pixels are in these portions of the context mask 310, the generative model will not generate pixels for these areas. As such, the areas of the masking 308 will not include any content. The area occupied by the original image 302 is also covered by the black pixels in the context mask, such that the content of the initial image will be preserved (e.g., not modified or recreated by the generative model). As such, the generative model will only generate pixels for the areas corresponding to the mask areas 312 and 314, thereby resulting in a reformatted image having the target aspect ratio.

[0067] As shown, the reformatted image 318 has a different aspect ratio than the initial image 302, and the content of the initial image 302 has been extended into the areas of the masks 312 and 314. Meanwhile, the content of the initial image 302 is maintained inside the dashed area 320, which corresponds to the area of the initial image 302. In this way, the original content of the initial image 302 can be maintained while also extending the content of the initial image 302 to create the reformatted image 318 having the target aspect ratio. The image conversion apparatus 175 can facilitate the insertion of the reformatted image 318 into the intended presentation area (e.g., target presentation area) as discussed later.

[0068] While changing an aspect ratio of the initial image is one way in which images can be reformatted, it is not the only way. For example, content of (e.g., objects within) the initial image 302 can be rearranged, relocated, or otherwise modified to create the reformatted image 320. For example, the instructions submitted to the generative model withAttorney Docket No. t>6113-087t>WOlthe context mask 310 can cause the generative model to move one or more objects to different locations of the reformatted image 318. More specifically, the generative model can be instructed to shift objects of the initial image closure to a right (or left) edge in the reformatted image 318. This is advantageous, for example, when third party content may be overlaid on the reformatted image 318 in a manner that would occlude objects of the reformatted image 318 without the shifting.

[0069] FIG. 4 is an illustration 400 of image masking and image reformatting using inpainting. The image conversion apparatus 175 is shown as performing the masking and reformatting. As previously discussed, the image conversion apparatus 175 can include a generative model that enables the reformatting, or the image conversion apparatus 175 can interact with one or more separate generative models to achieve the reformatting. For example, the image conversion apparatus 175 can be configured to perform all of the operations required to carry out the masking and reformatting, or the image conversion apparatus 175 can be configured to perform a portion of the operations (e.g., masking), while using a separate generative model to perform the reformatting.

[0070] As depicted by the illustration 400, the masking and reformatting are performed using an initial image 402. In some implementations, the initial image 402 is obtained by the image conversion apparatus 175. For example, the initial image 402 can be a fully rendered digital component in the form of a fixed image having one or more characteristics that are not directly editable without altering the original pixel data. For example, the initial image 402 can be an image having a given aspect ratio, such as 4:3, or another aspect ratio. The digital component may have been fully constructed and saved in an image format by a third party before being uploaded, or otherwise made accessible to the image conversion apparatus 175.

[0071] In the present example, the initial image 402 includes a user interface element 404. The user interface element 404 can be banner content, an interactive button, or another element that conveys information or provides interactive features. The user interface element 404 is positioned at a location that appears to occlude objects (e.g., the midsections of the people) depicted in the initial image 402. Because the initial image 402 is a fixed image, the image conversion apparatus 175 is generally unable to simply remove the user interface element 404 to reveal the occluded portions of objects occluded by the user interface elementAttorney Docket No. t>6113-087t>WOl404. Therefore, the image conversion apparatus 175 must generate the content occluded by the user interface element 404.

[0072] The image conversion apparatus 175 can generate the content occluded by the user interface element 404, for example, by masking the initial image 402, and using a generative model to perform inpainting to create a reformatted image 406. The image conversion apparatus 175 can insert a mask (e.g., masking pixels) 408 into the initial image 402 to signify where the generative model should create content pixels. In the present example, the image conversion apparatus 175 inserts the mask 408 in an area defined by the borders of the user interface element 404, thereby indicating that content pixels are to be created for an area of the initial image 402 at which the user interface element 404 is depicted.

[0073] The image conversion apparatus 175 can create a context mask 410 based on the masked version of the initial image 402. The masked version of the initial image 402 can also be referred to as a masked image. The context mask 410 includes a set of pixels (e.g., white pixels) indicating where content is to be generated, and a different set of pixels (e.g., black pixels) indicating areas to ignore or exclude from the generative process. For example, white pixels of the mask area 412 can indicate to the generative model that pixels should be generated to fill this area. Meanwhile, the black pixels in the mask area 414 can indicate to the generative model that this portion of the input image is to be ignored or excluded from the image generation process. Here, the area occupied by content of the initial image 402 surrounding the user interface element 404 is covered by the black pixels in the context mask 410, such that the content of the initial image surrounding the user interface element 404 will be preserved (e.g., not modified or recreated by the generative model). As such, the generative model will only generate pixels for the portion of the initial image 402 corresponding to the mask area 412, thereby resulting in a reformatted image 406 in which the user interface element 404 is replaced with content pixels generated based on the portions of the initial image 402 surrounding the user interface element 404.

[0074] For example, as shown, the generative model creates pixels that are within the dashed rectangle 416, and the user interface element 404 is not depicted in the reformatted image 406. As such, the reformatted image 406 can be used to create additional digital components that do not include the user interface element 404. For example, the reformattedAttorney Docket No. t>6113-087t>WOlimage 406 can be used to create digital components in which one or more different user interface elements are inserted and / or the content of the reformatted image is further reformatted or rearranged.

[0075] In some implementations, the image conversion apparatus 175 can determine that the reformatted image 406 needs to be reformatted further. As previously discussed, the image conversion apparatus 1 5 can determine that the reformatted image 406 needs to comply with one or more presentation requirements. For example, the image conversion apparatus 175 can determine that the current aspect ratio (e.g., set aspect ratio) of the reformatted image 406 differs from a target aspect ratio of a presentation area the image conversion apparatus 175 is populating. In this example, the current (set) aspect ratio of the initial image 302 can be 4:4 (e.g., a square), while the target aspect ratio of the presentation area may be 9: 16 (e.g., a vertical rectangle). In order for the image conversion apparatus 175 to use the reformatted image 406 to fully occupy the presentation area without significant distortion, the image conversion apparatus 175 must reformat the reformatted image 406. More specifically, the image conversion apparatus 175 needs to perform outpainting to create additional pixels at the top and bottom of the reformatted image 406, thereby extending the content of the reformatted image 406 in the vertical direction. This can be performed in substantially the same way as discussed above with reference to the reformatting of the initial image 302, and is therefore not repeated here.

[0076] FIG. 5A is an illustration of an example video streaming site user interface 500 for which the image reformatting can be used to create digital components for presentation. With reference to FIGs. 5A and 5B, the description will refer to operations being performed by the image conversion apparatus 175 to simplify the discussion, but it should be appreciated that some of the operations can also be performed by the 110 and / or that the image conversion apparatus 175 is implemented as part of the 110. For example, receiving a request, selecting content to provide in response to the request, and transmitting that content to a client device can be performed by the image conversion apparatus 175 or the 110 (as discussed above with reference to FIG. 1.

[0077] The user interface 500 includes a video presentation frame 502 in which video content 504 is displayed. In some implementations, digital components can be created for presentation near certain points during playback of the video content 504. For example, theAttorney Docket No. t>6113-087t>WOlvideo content 504 may be delineated into segments, as indicated by the segment end frames 506 and 508 in FIG. 5A. In this example, the image conversion apparatus 175 may reformat images for use in digital components that are presented at, or near, these segment end frames 506 and 508. Note that the specific frames for which digital components are generated can be selected in various different ways, such as at set times during playback of the video content 504.

[0078] Once a frame (e.g., a still image) of the video content 504 is selected for the creation of one or more digital components, the content of that frame can be extracted by the image conversion apparatus 175 as reference content for the reformatting of images used to create digital components. For example, the image conversion apparatus 175 can provide the reference content from the frame to the generative model with instructions to reformat an image to blend naturally with the color scheme or style of the reference content. In this way, the image conversion apparatus 175 can guide the reformatting of an image to visually integrate with and / or complement the content of the frame with which the digital component will be presented.

[0079] In some implementations, the image conversion apparatus 175 can determine a reference timestamp of the video at which the selected frame occurs in the video content 504. In these implementations, the image conversion apparatus 175 can store a reformatted image and / or digital component created using the reformatted image with a reference to the reference timestamp and / or the video content. For example, assume that the segment end frame 506 is presented 3:00 into playback of the video content 504. In this example, a reformatted image generated based on the segment end frame 506 can be stored with reference to (e.g., indexed to) the timestamp 3:00. As such, when the image conversion apparatus 175 receives a request for a digital component to be presented with the video content 504, the image conversion apparatus 175 can determine which digital component to select for presentation based on a current playback timestamp included in the request. In situations where the difference between current playback timestamp and the reference timestamp stored with a reformatted image and / or digital component generated using the reformatted image is within a specified threshold, the reformatted image (or digital component created using the reformatted image) can be selected for presentation. Otherwise, a different reformatted image or digital component can be selected for presentation.Attorney Docket No. t>6113-087t>WOl

[0080] FIG. 5B is another illustration of an example video streaming site user interface 500 for which the image reformatting can be used to create digital components for presentation. In FIG. 5B, the user interface 500 still includes the video presentation frame 502, but the video content 504 has been reduced in size and shifted to a side of the video presentation frame 502. This can occur, for example, when the video content 504 is paused and / or when a digital component 506 is presented in the video presentation frame 502 at the same time as the video content 504.

[0081] To illustrate, assume that the video content 504 was paused, and a request for a digital component was submitted to in response to the pausing. In this example, the image conversion apparatus 175 can determine the current playback timestamp and select the digital component 506 having a reference timestamp that is within a specified threshold (e.g., 1 second, 3 seconds, 5 seconds, 10 seconds, or another specified amount of time) of the reference timestamp assigned to the digital component 506. The digital component 506 is depicted as including the content of the reformatted image 318 discussed with reference to FIG. 3 and the text content “Need a vacation?”. As shown, the digital component 506 has the vertical rectangular aspect ratio, as previously discussed, and is presented adjacent to the paused / smaller video playback depiction.

[0082] As previously discussed, the reformatted image 318 and / or the digital component 506 can be generated based on the presentation area available for presentation of a digital component while the video content 504 is still presented. In the present example, there are two options for presentation areas. The first presentation area is an adjacent presentation area 508 that is represented by the solid rectangle surrounding the digital component 506. A second presentation area can be a backdrop presentation area 510 that functions as a backdrop to the video content 504 within the video presentation frame 502. That is, the content of a reformatted image and / or digital component that is presented in the backdrop presentation area 510 will have content that is adjacent to / abutting at least two edges of the area presenting the video content 504. This is visually depicted by the suns 512 that are depicted above and below the video content 504.

[0083] To generate a reformatted image that fills the backdrop presentation area 510, complements the video content 504, and does not have meaningful content occluded by the video content 504 when presented in the presentation area 510, the image conversionAttorney Docket No. t>6113-087t>WOlapparatus 175 can use the dimensions of the backdrop presentation area 510 when reformatting an initial image. For example, the image conversion apparatus 175 can create a context mask having an aspect ratio of the backdrop presentation area 510 to instruct the generative model as to the aspect ratio of the area the reformatted image needs to occupy. However, the image conversion apparatus 175 can use the area occupied by the video content 504 in FIG. 5B as a guide for inserting a set of pixels (e.g. black pixels) indicating that the generative model should ignore, or exclude from the generative process, the area in which the video content 504 is presented. In this way, meaningful content that conveys information to the viewer of the user interface 500 of FIG. 5B will not be occluded by the video content 504, and the generative model can focus the arrangement of meaningful objects in the reformatted image in the areas that are not occupied by the video content 504.

[0084] FIG. 6 is a flow chart of an example process 600 for reformatting images.Operations of the process 600 can be performed by one or more systems that include one or more computing devices. For example, operations of the process 600 can be performed by the 110, the image conversion apparatus 175, or both. Operations of the process 600 can be implemented as instructions stored on one or more non-transitory computer readable medium, and execution of the instructions can cause specially configured and / or programmed circuitry to perform operations of the process 600.

[0085] An initial image is obtained (602). In some implementations, the initial image has one or more fixed attributes, such as a fixed / set aspect ratio, fixed color scheme, or depicts a fixed set of objects. The initial image can be an image that was uploaded by a third party in an image format, such that the initial image is fully constructed prior to being obtained.

[0086] A determination is made that the initial image should be reformatted (604). The determination can include determining that the fixed / set aspect ratio of the initial image differs from a target aspect ratio of a target presentation area. For example, as discussed above, the initial image may have a square aspect ratio or a horizontal rectangle aspect ratio, while the aspect ratio required by the target presentation area may be a vertical rectangle aspect ratio. In this scenario, the aspect ratio of the initial image does not comply with the aspect ratio required by the target presentation area.

[0087] Other determinations that the initial image needs to be reformatted can include, for example, a determination that other content will be overlaid on a portion of the imageAttorney Docket No. t>6113-087t>WOlwhen presented in the target presentation area. In this example, the content being overlaid on the image may require objects depicted by the image to be rearranged, or otherwise modified to avoid meaningful content from being occluded during presentation of the image with the other content. For example, as discussed above with reference to FIG. 5B, the determination can be made that the area in which the video content 504 is presented will overlap an image / digital component presented in the backdrop presentation area 510, thereby requiring reformatting of the image.

[0088] A still image is obtained 606. In some implementations, the still image is of a video with which a reformatted image will be presented. For example, as discussed above with reference to FIGs. 5A-5B, the still image can be a frame of a video with which the reformatted image and / or a digital component created using the reformatted image will be presented. In other implementations, the still image can be an image of other content (e.g., an application user interface or a web page user interface) with which the reformatted image will be presented. As previously discussed, the obtained image can be provided to the generative model with the initial image to guide reformatting of the initial image. For example, the generative model can reformat the initial image based on a color scheme of the obtained image, textures of the obtained image, fonts of text used in the obtained image, or other attributes of the obtained image.

[0089] The initial image is masked with masking pixels (608). In some implementations, the masking pixels at indicate where additional image content (e.g., pixels) is required to be generated and / or areas of the initial image that should be ignored / not processed during the generative process. The masking results in a masked image that includes pixels of the initial image and the masking pixels. As previously discussed, different color pixels can be used to differentiate between areas where additional / new pixels should be generated and other areas where additional / new pixels should not be generated. For example, black pixels can be used to define exclusion zones where generative content should not be included / inserted, and white pixels can be used to define inclusion zones where generative content should be included / inserted.

[0090] A reformatted image is generated (610). In some implementations, the reformatted image has at least one different visual characteristic relative to the initial image. For example, as previously discussed, the reformatted image can have a target aspect ratio ofAttorney Docket No. t>6113-087t>WOlthe target presentation area instead of the fixed / set aspect ratio of the initial image. The reformatted image can be generated based on application of a generative model to the masked image.

[0091] During the generative process, new pixels are generated for the reformatted image based on content of the initial image and content of the obtained still image. For example, as previously discussed, the obtained still image and the masked image can be input to the generative model with instructions that cause the generative model to create the reformatted image. The instructions can specify how the generative model should use the obtained image, how objects in the reformatted image should be arranged, and / or how the reformatted image should visually appear relative to the still image with which the reformatted image will ultimately appear in a user interface. For example, the instructions can specify that the reformatted image should use a color scheme that blends with the color scheme of the still image. The reformatting can include rearranging visual features of the initial image based on a location of a frame within which the video is visually presented during a pause of the video, as discussed above with reference to FIG. 5B.

[0092] A reference timestamp can be determined (612). In some implementations, the reference timestamp can be a timestamp of a video at which the obtained still image appears. For example, as previously discussed, if the obtained still image is a frame of a video that is presented 3:00 into playback of the video, the reference timestamp can be determined to be 3 :00. To determine the reference timestamp, the time presented by a playback timer of a video playback program / application can be extracted when the frame is presented.

[0093] The reformatted image is stored with a reference to the reference timestamp (614). In some implementations, the reference to the reference timestamp is the value of the reference timestamp. Continuing with the example above, the value 3:00 can be assigned to the reformatted image specifying that the reformatted image was created based on the frame appearing at 3:00 into playback of the video.

[0094] A request for content is received (616). In some implementations, the request can be for an image and / or a digital component. For example, the reformatted image may be combined with other objects, such as interactive buttons or text overlaid on the image after the reformatting, such that the reformatted image is further transformed into a digital component that includes content that was not present at the time the reformatted image wasAttorney Docket No. t>6113-087t>WOlcreated. For example, the request can be a request that the system provide a digital component for presentation with a video being played at a client device. The request can include various information, similar to the information included in the requests discussed with reference to FIG. 1.

[0095] The request can also include an identifier of the video and / or a current playback timestamp. The current playback timestamp can indicate, for example, the value of the playback timer at the time the request was sent and / or when the target presentation area will be available for presentation of the requested content. For example, the target presentation area may be available to present content when the video reaches 3:00 into playback of the video, but the request may be sent prior to that to allow time for the content to be selected based on the request and transmitted to the client device.

[0096] A time difference between the current playback timestamp and the reference timestamp can be determined (618). In some implementations, the time difference is determined by subtracting the current playback timestamp from the reference timestamp, with the resulting value being the time difference.

[0097] The reformatted image is inserted into the target presentation area corresponding to the obtained still image (620). In some implementations, the inserting results in a visual presentation that includes both of the reformatted image (and / or digital component created using the reformatted image) and the video (e g., the obtained still image). For example, as discussed with reference to FIG. 5B, the reformatted image can be presented adjacent to, or as a backdrop around, the video content. Other presentation locations can also be used to present the reformatted image with the video content. In some implementations, the insertion of the reformatted image into the presentation area is performed based on the time difference between the current playback timestamp and the reference timestamp being less than a specified threshold.

[0098] FIG. 7 is a flow chart of an example process 700 for reformatting images.Operations of the process 700 can be performed by one or more systems that include one or more computing devices. For example, operations of the process 700 can be performed by the 110, the image conversion apparatus 175, or both. Operations of the process 700 can be implemented as instructions stored on one or more non-transitory computer readableAttorney Docket No. t>6113-087t>WOlmedium, and execution of the instructions can cause specially configured and / or programmed circuitry to perform operations of the process 700.

[0099] A reference image is obtained (702). In some implementations, the reference image is an image included in a set of reference images that have been selected based on visual quality and / or other characteristics. For example, the set of reference image can be a set of images that have been classified as images having high visual quality scores, such as scores indicating that the image has a specified level of clarity, brightness, contrast, and / or other visual qualities. The set of reference images can also be identified, at least in part, based on historical user interactions with the images. For example, each of the images can be scored based on a level of user interaction with the image (e.g., a number of users clicks of the image) when presented as a digital component. The images classified as reference images can be required to have at least a specified level of user interactions. The set of reference images can be pre-existing (e.g., stored in a database), such that the reference image can simply be obtained from a database.

[0100] The reference image is masked to create a training image (704). The reference image can be masked to create an inpainting training image or an outpainting training image, as discussed with reference to FIG. 2. As previously discussed, masking the reference image can include masking a first set of pixels of the reference image indicating where additional image content is required to be generated. For example, the masking can include padding the reference image with a border ratio in an outpainting training cycle. In other words, a portion of the edges of the reference image can be masked with pixels indicating areas of the reference image that should be ignored by the generative model. Those ignored pixels will be generated by the generative model, and the generated pixels will be compared to the ignored pixels of the reference image to evaluate the model error.

[0101] In an inpainting training cycle, masking the reference image includes masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image. For example, the reference image can be masked with at least one of (i) two random non-overlapping bounding boxes or (ii) one horizontally rectangular bounding box. The locations of the masked can be varied to create more training images.

[0102] A generative model is applied to the training image to create a corresponding test image (706). For instance, the model can take the masked image as input and generateAttorney Docket No. t>6113-087t>WOlcontent for the masked portion of the masked image, and output a test image. More specifically, the training image can be input into the generative model with model instructions that cause the generative model to create an output image using the training image. As previously discussed, the generative model will generate pixels based on the mask pixels.

[0103] An error value is generated (708). In some implementations, the error value represents differences between the training image and the corresponding test image created using the training image. The error value can be determined based on differences between (i) the additional content generated at locations of the first set of pixels and (ii) reference content of the reference image. As previously discussed, there are several techniques for computing the error value, and those techniques are not repeated here.

[0104] Parameters of the generative model are adjusted based on the error value (710). In some implementations, the model parameters can be adjusted in the direction that reduces the loss (e.g., the error value). Generally speaking, this can be performed by computing gradients that indicate how much each model parameter contributed to the error, and subtracting a small portion of the gradient from the current weights of the model parameters. The generation of test images and adjustment of model parameters can iterate until a stop condition is reached. At that point, the generative model can be deployed to reformat images.

[0105] FIG. 8 is a block diagram of an example computer system 800 that can be used to perform operations described above. The system 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. Each of the components 810, 820, 830, and 840 can be interconnected, for example, using a system bus 850. The processor 810 is capable of processing instructions for execution within the system 800. In one implementation, the processor 810 is a single-threaded processor. In another implementation, the processor 810 is a multi -threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 or on the storage device 830.

[0106] The memory 820 stores information within the system 800. In one implementation, the memory 820 is a computer-readable medium. In one implementation, the memory 820 is a volatile memory unit. In another implementation, the memory 820 is a non-volatile memory unit.Attorney Docket No. t>6113-087t>WOl

[0107] The storage device 830 is capable of providing mass storage for the system 800. In one implementation, the storage device 830 is a computer-readable medium. In various different implementations, the storage device 830 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0108] The input / output device 840 provides input / output operations for the system 800. In one implementation, the input / output device 840 can include one or more of a network interface device, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and / or a wireless interface device, e.g., and 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other peripheral devices, e.g., keyboard, printer and display devices 860. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0109] Although an example processing system has been described in FIG. 8, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0110] An electronic document (which for brevity will simply be referred to as a document) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0111] For situations in which the systems discussed here collect and / or use personal information about users, the users may be provided with an opportunity to enable / disable or control programs or features that may collect and / or use personal information (e.g., information about a user’s social network, social actions or activities, a user’s preferences or a user’s current location). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information associated with the user is removed. For example, a user’s identity may be anonymized so that the no personally identifiable information can be determined for the user, or a user’s geographic location mayAttorney Docket No. t>6113-087t>WOlbe generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.

[0112] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0113] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0114] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtimeAttorney Docket No. t>6113-087t>WOlenvironment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0115] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0116] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0117] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), aAttorney Docket No. t>6113-087t>WOlmobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0118] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0119] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).Attorney Docket No. t>6113-087t>WOl

[0120] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0121] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0122] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0123] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. InAttorney Docket No. t>6113-087t>WOladdition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0124] What is claimed is:

Claims

Attorney Docket No. t>6113-087t>WOlCLAIMS1. A method, comprising:obtaining, by a system including electrical circuitry, an initial image having a set aspect ratio;determining, by the system, that the set aspect ratio of the initial image differs from a target aspect ratio of a target presentation area;obtaining, by the system, a still image of a video with which the target presentation area is presented;masking, by the system, the initial image with masking pixels indicating where additional image content is required to be generated, wherein the masking results in a masked image that includes pixels of the initial image and the masking pixels;generating, by the system, a reformatted image having the second aspect ratio of the target presentation area based on application of a generative model to the masked image, including generating new pixels for the reformatted image based on content of the initial image and content of the still image; andinserting, by the system, the reformatted image into the target presentation area resulting in a visual presentation that includes both of the reformatted image and the video.

2. The method of claim 1, further comprising training the generative model based on a set of training images, including:obtaining a reference image;masking the reference image, wherein the masking of the reference image creates a training image;applying the generative model to the training image, wherein application of the generative model to the training image creates a corresponding test image; and determining an error value representing differences between the reference image and the corresponding test image created using the training image; andadjusting parameters of the generative model based on the error value.

3. The method of claim 2, wherein:masking the reference image comprises masking a first set of pixels of the referenceAttorney Docket No. t>6113-087t>WOlimage indicating where additional image content is required to be generated; and determining the error value representing differences between the training image and the corresponding test image created using the training image comprises determining the error value based on differences between (i) the additional content generated at locations of the first set of pixels and (ii) reference content of the reference image.

4. The method of claim 3, wherein masking the reference image comprises: padding the reference image with a border ratio in an outpainting training cycle; and masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle.

5. The method of claim 4, wherein masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle comprises masking the reference image with at least one of (i) two random non-overlapping bounding boxes or (ii) one horizontally rectangular bounding box.

6. The method of claim 1, further comprising:determining a reference timestamp of the video at which the still image of the video appears;storing the reformatted image with a reference to the reference timestamp; receiving a request for an image, wherein the request identifies the video and includes a current playback timestamp; anddetermining a time difference between the current playback timestamp and the reference timestamp, wherein:inserting the reformatted image into the target presentation area is performed based on the time difference being less than a specified threshold.

7. The method of claim 1, further comprising rearranging visual features of the initial image based on a location of a frame within which the video is visually presented during a pause of the video.Attorney Docket No. t>6113-087t>WOl8. A non-transitory computer readable medium storing instructions that, upon execution by one or more data processing apparatus including electrical circuitry, cause the one or more data processing apparatus to perform operations comprising:obtaining an initial image having a set aspect ratio;determining that the set aspect ratio of the initial image differs from a target aspect ratio of a target presentation area;obtaining a still image of a video with which the target presentation area is presented; masking the initial image with masking pixels indicating where additional image content is required to be generated, wherein the masking results in a masked image that includes pixels of the initial image and the masking pixels;generating a reformatted image having the second aspect ratio of the target presentation area based on application of a generative model to the masked image, including generating new pixels for the reformatted image based on content of the initial image and content of the still image; andinserting the reformatted image into the target presentation area resulting in a visual presentation that includes both of the reformatted image and the video.

9. The non-transitory computer readable medium of claim 8, wherein the instructions cause the one or more data processing apparatus to perform operations further comprising:training the generative model based on a set of training images, including:obtaining a reference image;masking the reference image, wherein the masking of the reference image creates a training image;applying the generative model to the training image, wherein application of the generative model to the training image creates a corresponding test image; andAttorney Docket No. t>6113-087t>WOldetermining an error value representing differences between the reference image and the corresponding test image created using the training image; andadjusting parameters of the generative model based on the error value.

10. The non-transitory computer readable medium of claim 9, wherein: masking the reference image comprises masking a first set of pixels of the reference image indicating where additional image content is required to be generated; and determining the error value representing differences between the training image and the corresponding test image created using the training image comprises determining the error value based on differences between (i) the additional content generated at locations of the first set of pixels and (ii) reference content of the reference image.

11. The non-transitory computer readable medium of claim 10, wherein masking the reference image comprises:padding the reference image with a border ratio in an outpainting training cycle; and masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle.

12. The non-transitory computer readable medium of claim 11, wherein masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle comprises masking the reference image with at least one of (i) two random non-overlapping bounding boxes or (ii) one horizontally rectangular bounding box.

13. The non-transitory computer readable medium of claim 8, wherein the instructions cause the one or more data processing apparatus to perform operations further comprising:determining a reference timestamp of the video at which the still image of the video appears;storing the reformatted image with a reference to the reference timestamp; receiving a request for an image, wherein the request identifies the video and includesAttorney Docket No. t>6113-087t>WOla current playback timestamp; anddetermining a time difference between the current playback timestamp and the reference timestamp, wherein:inserting the reformatted image into the target presentation area is performed based on the time difference being less than a specified threshold.

14. The non-transitory computer readable medium of claim 8, wherein the instructions cause the one or more data processing apparatus to perform operations further comprising rearranging visual features of the initial image based on a location of a frame within which the video is visually presented during a pause of the video.

15. A system, comprising:one or more memory devices; andone or more data processing apparatus, including electronic circuitry, configured to access the one or more memory devices and execute instructions that cause the one or more data processing apparatus to perform operations comprising:obtaining an initial image having a set aspect ratio;determining that the set aspect ratio of the initial image differs from a target aspect ratio of a target presentation area;obtaining a still image of a video with which the target presentation area is presented;masking the initial image with masking pixels indicating where additional image content is required to be generated, wherein the masking results in a masked image that includes pixels of the initial image and the masking pixels;generating a reformatted image having the second aspect ratio of the target presentation area based on application of a generative model to the masked image, including generating new pixels for the reformatted image based on content of the initial image and content of the still image; andinserting the reformatted image into the target presentation area resulting in a visual presentation that includes both of the reformatted image and the video.Attorney Docket No. t>6113-087t>WOl16. The system of claim 15, wherein the instructions cause the one or more data processing apparatus to perform operations further comprising:training the generative model based on a set of training images, including:obtaining a reference image;masking the reference image, wherein the masking of the reference image creates a training image;applying the generative model to the training image, wherein application of the generative model to the training image creates a corresponding test image; and determining an error value representing differences between the reference image and the corresponding test image created using the training image; andadjusting parameters of the generative model based on the error value.

17. The system of claim 16, wherein:masking the reference image comprises masking a first set of pixels of the reference image indicating where additional image content is required to be generated; and determining the error value representing differences between the training image and the corresponding test image created using the training image comprises determining the error value based on differences between (i) the additional content generated at locations of the first set of pixels and (ii) reference content of the reference image.

18. The system of claim 17, wherein masking the reference image comprises: padding the reference image with a border ratio in an outpainting training cycle; and masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle.

19. The system of claim 18, wherein masking the reference image at one or more locations that are at least a threshold distance from edges of the reference image in an inpainting training cycle comprises masking the reference image with at least one of (i) two random non-overlapping bounding boxes or (ii) one horizontally rectangular bounding box.Attorney Docket No. t>6113-087t>WOl20. The system of claim 15, wherein the instructions cause the one or more data processing apparatus to perform operations further comprising:determining a reference timestamp of the video at which the still image of the video appears;storing the reformatted image with a reference to the reference timestamp; receiving a request for an image, wherein the request identifies the video and includes a current playback timestamp; anddetermining a time difference between the current playback timestamp and the reference timestamp, wherein:inserting the reformatted image into the target presentation area is performed based on the time difference being less than a specified threshold.

Images