Streaming system and method

Abstract

A method of streaming for rendered images by a server, wherein the rendered images are generated in a multi-stage rendering pipeline, the method comprises the steps of, for a next image following a preceding image, generating first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image; and transmitting the first motion vectors to a client device before rendering of the next image is complete, thereby enabling the client device to construct an approximation of the next image using the preceding image and the first motion vectors.

Description

PATENT Attorney Docket No. 116457-1540087 Client Ref. No. SYP356373WO01STREAMING SYSTEM AND METHOD CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This International Patent Application claims priority to United Kingdom Patent Application No. 2500503.4, filed on January 15, 2025, the disclosure of which is herein incorporated by reference in its entirety for all purposes.FIELD OF THE INVENTION

[0002] The present invention relates to a streaming system and method.DESCRIPTION OF THE PRIOR ART

[0003] Conventionally, videogames have been played using PCs or consoles that generate the game content locally. However, recently so-called cloud gaming or game streaming has become popular. In this case, the game is executed in the cloud, and streamed to the player’s client device. Notably therefore the client device itself need not be capable of natively executing the game, broadening access to it.

[0004] However, one issue with game streaming is the potential for lag or delay between user inputs for upload to the cloud server, and updated game images streamed down from the cloud server.

[0005] The present invention seeks to address or mitigate this problem.SUMMARY OF THE INVENTION

[0006] Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description.

[0007] In a first aspect, a method of streaming for rendered images by a server is provided in accordance with claim 1.

[0008] In another aspect, a method of displaying streamed rendered images for a client is provided in accordance with claim 8.

[0009] In another aspect, a server for streaming rendered images is provided in accordance with claim 12.

[0010] In another aspect, a client device for displaying streamed rendered images is provided in accordance with claim 14.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:Figure 1 is a schematic diagram of an entertainment device suitable as a client or server in accordance with embodiments of the description.Figure 2 is a schematic diagram of a streaming system for interactive content in accordance with embodiments of the description.Figure 3 is a schematic diagram of a rendering pipeline in accordance with embodiments of the description.Figures 4A-D are schematic diagrams of image data transmissions in accordance with embodiments of the description.Figure 5 is a flow diagram of a method of streaming for a server in accordance with embodiments of the description.Figure 6 is a flow diagram of a method of displaying streamed images for a client in accordance with embodiments of the description.DESCRIPTION OF THE EMBODIMENTS

[0012] A streaming system and method are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.

[0013] Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 shows an example of an entertainment system 10 such as a computer or console.

[0014] Depending on the role it is configured for (e.g. by suitable software instruction), the entertainment device may operate either as a cloud game server, or as a client device, and is used as an example for both herein.

[0015] The entertainment system 10 comprises a central processor or CPU 20. The entertainment system also comprises a graphical processing unit or GPU 30, and RAM 40. Two or more of the CPU, GPU, and RAM may be integrated as a system on a chip (SoC). Further storage may be provided by a disk 50.

[0016] The entertainment device may transmit or receive data via one or more data ports 60. It may also optionally receive data via an optical drive 70. Audio / visual outputs from the entertainment device are typically provided through one or more A / V ports 90 or one or more of the data ports 60. Where components are not integrated, they may be connected as appropriate either by a dedicated data link or via a bus 100.

[0017] Examples of a device for displaying images output by the entertainment system include a head mounted display ‘HMD’ 120 worn by a user 1, a TV (not shown), and a portable screen 140.

[0018] Interaction with the system is typically provided using one or more handheld controllers 130, 140, and / or one or more VR controllers (130A-L,R) in the case of the HMD.

[0019] Referring now to Figure 2, entertainment device 10, in cooperation with a controller such as handheld controller 130, communicates with a cloud server 200 via a network 150 such as the internet. The entertainment device receives input signals from the controller, and relays them (or a parsed version of them, for example input IDs and values rather than raw signals) to the cloud server 200. The cloud server receives these inputs and uses them in the game being run on the server, typically as if the inputs were conventional local inputs so that the game can operate transparently. The output of the cloud server (video images and typically also audio) are then streamed back to the entertainment device via the network. Similarly a device 140 with built-in controller inputs can interact with the server in a similar manner.

[0020] However, the round trip in the network from the entertainment device to the cloud server, and from the cloud server to the entertainment device, can introduce lag compared to what would be experienced during local play. Typically, the round trip network latency for the player input and the server output may be in the order of 10-20 milliseconds each way.

[0021] This may seem small, but typically videogames aim to run at 60 frames per second in order to feel responsive for the user. Consequently there is normally only a 16 millisecond period between frames during which the game state is updated, and therefore a variable round trip delay of a similar order of time can significantly affect the apparent responsiveness of the game.

[0022] Accordingly, in embodiments of the present description, a system and method are provided to reduce the effective round trip delay by helping the client device to construct at least an approximate version of the next image earlier than would otherwise be the case.

[0023] A known technique for efficiently transmitting images is to send a first image (via any suitable method), and then when sending a next image, represent this using so-called motion vectors that describe how to approximately construct the next image by shifting blocks of the first image to new positions. Then, any residual ‘error’ between the constructed next image and the actual next image can optionally be transmitted as a smaller set of supplementary data.

[0024] Hence for example, a simple horizontal pan of an image may result in all the blocks of the first image moving left (or right) in the same direction and by the same amount, and the residual ‘error’ is just the strip of new image that has appeared on the right (or left) hand edge of the image. This is an efficient way to transmit an image.

[0025] For more complex image sequences, for example zooming in or moving forward, the motion vectors will typically radiate out from the centre of zoom or motion. Meanwhile objects within the scene that move will have additional components of their motion vectors related to their in-scene motion. In these cases, the motion vectors may produce a very good approximation of the next image if all the components of it existed in the previous image, which is often the case.

[0026] Summarising, then typically there will be components of the motion vectors that are due to the image viewpoint (e.g. x, y, or z motion, and zoom) and components of the motion vectors that are due to motion within the image (e.g. people walking, cars moving, etc). The motion vectors can be applied to blocks (e.g. blocks of pixels 2x2, 4x4, 8x8, 16x16, or any suitable size), and the blocks themselves need not necessarily be the same size or shape.

[0027] The transmission process therefore comprises obtaining the first image, transmitting it, obtaining a second image, calculating how to approximately construct it from blocks of the first image, transmitting the resulting motion vectors, and optionally calculating the difference between the constructed second image and the actual second image, and then transmitting data representing that difference as error correction data. This is then repeated for subsequent images.

[0028] Whilst this is an efficient method, it is still subject to the round trip network latency mentioned previously, and also subject to any computational bottleneck in the client device, which must use the motion vectors to reconstruct the second image and apply the error corrections in a timely manner to output the resulting image.

[0029] Embodiments of the present description seek to address or mitigate this problem, by exploiting the fact that in a videogame, the image is constructed in a series of stages. The stages are part of what is known as a rendering pipeline, the output of which is an image that can then be transmitted from the server to the client, for example using the techniques described above.

[0030] Referring now to Figure 3, after the game logic (game state) has been updated to reflect the current situation in-game that is to be depicted in the image, the pipeline 300 comprises some or all of the following (and potentially other steps not included here), in roughly, but not necessarily, the order given below.

[0031] Camera update (310)The position and / or orientation of the virtual camera within the environment may change, for example in response to movement of the player’s character and either an automatic tracking camera (in the case of so-called third-person games) or motion and directional input (in the case of so-called first person games). Meanwhile some games have a fixed camera (e.g. 2D platform games) and so this update is skipped or a minimal process.

[0032] Geometry update (320)The geometry of the environment to be rendered may be updated if there are new elements to show, or if elements have changed. Similarly, changes in character animation / pose geometry may be defined / updated.

[0033] Z-buffer update (330)What aspects of the environment are visible is updated depending on the geometry and the virtual camera viewpoint - parts that are unseen need not be processed, or only processed to a limited extent.

[0034] Texturing (340)Texture maps are used to apply basic graphical imagery to the geometry of the scene. Typically different textures are used as a function of distance from the viewpoint.

[0035] Static Lighting (350)Lightmaps may be used for direct and indirect lighting, shadows, and reflections.

[0036] Dynamic lighting (360)Direct and indirect lighting, shadows, and reflections may be computed for dynamic ingame light sources such as a player’s torch.

[0037] Atmospherics (370)Effects such as mist or fog, particle effects, translucency (water) and the like.

[0038] Post processing (380)A variety of effects that may be applied, often to the image as a whole, including bloom, exposure, depth of field bokeh, lens effects, and the like.

[0039] Upscaling (390)If, in order to achieve a preferred frame rate, the rendering has been performed at a lower resolution than intended, some systems have a machine learning based Al upscaling step to in-paint the rendered image up to the target resolution. Optionally this step may be implemented at the client device, after receiving a lower resolution version of the image.

[0040] Whilst these stages are illustrated at equal size in Figure 3, in practice some stages take longer than others, and some may overlap or be revisited during the rendering process. Similarly, as noted above some stages may occur in a different order to that listed. However, in general the image is formed using successive stages.

[0041] As discussed previously, motion vectors for a subsequent image typically comprise components related to the viewpoint of the camera, and components related to the movement of objects within the scene.

[0042] In embodiments of the present description, the components can be estimated after stages 310 and 320, since at this point both the change in camera viewpoint and the changes to objects (represented by their geometry) in the next image is already known, despite the image itself not yet existing.

[0043] Optionally the estimation may occur after stage 330, z-culling, as this estimates what elements of the objects are visible in the new image.

[0044] In any event, motion vectors may be calculated based on any motion of the virtual camera, as described elsewhere herein, and any motion of the geometry. The motion of the geometry may be calculated from the change of position of respective vertices of the polygons representing the geometry of the scene within the coordinate system of the game world.

[0045] Depending on how the specific rendering pipeline works, optionally the motion of the geometry may be calculated from the change of position of respective vertices of the polygons representing the geometry of the scene within a coordinate system of the virtual camera, thereby combining the components of the viewpoint and object motion.

[0046] In any event, a set of motion vectors can thus be generated for the next image early on in the rendering pipeline, and transmitted to the client.

[0047] Referring to Figure 4A, this illustrates the conventional process of transmitting images. Rendering, encoding, and display delays common to all cases are omitted for clarity. Meanwhile all timings in relation to Figures 4A to 4C are non-limiting and exemplary only.

[0048] At a notional time of 0ms, the current input available from the user is used to update the game state. Then (as a non-limiting example) the image representing the current view of that game state is generated over the next 16ms as a first image 310, and is transmitted to a client, taking a further 15ms. The client then displays the image 310. The total delay from updating the game state to displaying it at the client is thus 16+15 = 31ms.

[0049] Meanwhile, the second image 320 is generated between 16ms and 32ms, representing the new game state based on current inputs. This time, once the image has been generated, motion vectors are computed, a reconstruction of the image is created, this is compared to the original image, error correction data is computed, and then the motion vectors and error data (‘[M+E]’ in Figure 4A) are transmitted. The client then builds and correct (‘[B+C]’ in Figure 4A) a version of the image 320’ from this data.

[0050] Assuming that the generation of the motion vectors and error data takes a further 2ms (as a non-limiting example), the gain is that the smaller amount of data may take less time to reach the client (for example 10ms), whilst the building and correcting at the client may also take 2ms, so that overall the downstream delay is 2+10+2 = 14ms rather than 15ms, and the total delay from updating the game state to displaying it at the client is thus 14+16 = 30ms rather than 15+16 = 31ms, thereby reducing the delay by 1ms.

[0051] This reduced delay carries forward to subsequent images, so that e.g. image 330 also only has a transmission delay of 12ms until it is reconstructed as image 330’ again with a total delay of 30ms.

[0052] It will be appreciated that if the normal transmission delay of 15ms is longer, the benefits of using motion vectors may be more pronounced.

[0053] By contrast, referring now to Figure 4B, embodiments of the present description operate differently.

[0054] The first image may be transmitted in the same way as the first image in Figure 4A, e.g. transmitting an encoded version of the complete image, with a transmission delay of 15ms and a full delay of 31ms.

[0055] Now, however, the system computes motion vectors from the changes in camera viewpoint and / or geometry, once they have been updated within the rendering pipeline, as described elsewhere herein e.g. with reference to Figure 3.

[0056] In this case the changes may have been computed after, as a non-limiting example, 3ms out of the total 16ms for generating the frame, and the motion vectors take 1ms to compute (since there is no error data calculation at this stage, the process is shorter). The motion vector data can then be sent to the client immediately. Because it is smaller than the motion and error data in Figure 4A, it may arrive faster, e.g. in 7ms. The building of the image by the client may take 1ms (with no additional error correction) for a total downstream delay of 3+1+7+1 = 12ms. Clearly this compares very favourably with the total transmission delay of 31ms or 30ms as provided previously.

[0057] In this example, potentially the motion vectors for the second image arrive before the first image is completely received, but this is not significant. If this occurs, the client would wait until the first image is complete before attempting to compute the second image using the first image and the motion vectors.

[0058] However, it will be clear that a version of the second image 320”, based on motion vectors and no error correction is available quickly, and could in principle be displayed with an effective zero transmission lag at the 32ms time point, since 12ms is less than the 16ms window for generating the new image (denoted by the dashed line in the gap between the end of building ‘[B]’ and displaying image 320” at 32ms). Clearly this is a significant improvement in the apparent responsiveness of the game.

[0059] However, this responsiveness may come at a cost to graphical fidelity, as the resulting reconstructed image may not be perfect as it is built from elements of the preceding image using motion vectors that only relate to the geometry and camera viewpoint. Whilst these may represent the majority of motion in the image they do not, for example, represent non-geometric movement such as some shadows, lighting, reflections, particles and other post-processing effects in the image.

[0060] This means that some motion within the final image might not be captured by this early motion vector data.

[0061] Accordingly, and referring now also to Figure 4C, in embodiments of the present description once the image has reached a predetermined point in the rendering pipeline where it is representative of the final image (e.g. after post-processing, but optionally before any upscaling), e.g. at 32ms, a new set of image-based motion vectors [M’] are computed, e.g. in a similar manner to the conventional computation of motion vectors, e.g. using optical flow or similar between the current and one or more prior images. In this case the motion vectors again may be per-pixel or may be at a lower resolution (e.g. blocks of pixels 2x2, 4x4, 8x8, 16x16, or any suitable size), and the blocks themselves need not necessarily be the same size or shape, or a sub -sampling such as 1 pixel within 2x2 pixels, or 1 pixel within 4x4 pixels (or 2x2 pixels within 4x4 pixels, etc).

[0062] Notably, these new motion vectors are then compared with the equivalent motion vectors previously generated using the camera / geometry changes. Where the motion vectors match (optionally to within a predetermined tolerance), then it can be assumed that those parts of the reconstructed image at the client will remain the same; meanwhile where the motion vectors do not match (optionally outside a predetermined tolerance), then it can be assumed that the image has changed further due to non-geometric and non-camera based influences. Therefore, just the motion vectors for those parts of the image can now be transmitted.

[0063] This significantly reduces the amount of motion vector data to be transmitted once image generation has been completed, which in turn reduces the transmission delay again, e.g. to 5ms (again as a non-limiting example). The residual motion vector data can be used to partially re-build [B’] the image in response.

[0064] Hence in Figure 4C (and only showing the scenario for image 320 for simplicity), then as before a version 320” of image 320 can optionally be available immediately at 32ms for an effective transmission delay of 0ms and a total delay of 16ms, and updated motion vectors for parts of the image that need them can be received at the client after a downstream delay of 1+5+1 = 7ms and a total delay of 7+16 = 23ms. Again this is an improvement over the total delays of 31ms and 30ms discussed previously, particularly if the original new image 320” is displayed prior to the updated new image 320” ’ .

[0065] Finally, referring now to Figure 4D, optionally error data ‘[E]’ can be computed. The error data may be optional for some images or may be necessary for others, e.g. where panning introduces new image elements. In any event, the error data may be computed (e.g. for another1ms) and then transmitted (e.g. for 5ms) before being used to correct ‘[C]’ the image at the client (e.g. for another 1ms). Hence whilst shown as separate or separable steps in Figure 4D, this is similar to the [M+E] and [B+C] steps of Figure 4A. This is so that in principle image 320”’ can be displayed before image 320lvis displayed, after l+l+5+l=8ms and a total delay of 8+16=24ms. Again this is an improvement over the total delays of 31 or 30ms discussed previously, for an image of equivalent quality.

[0066] Accordingly, in embodiments of the present description, motion vectors are generated before all the stages of the rendering pipeline are complete, and typically after one of updating the camera viewpoint (if relevant), updating the geometry (if relevant) and updating the z-buffer (if relevant). It will be appreciated that for some games, such as 2D platformers, the camera viewpoint may not change, and the geometry may be replaced either wholly or in part by updates to sprites and / or sprite positions. Hence in this case the motion vectors may be similarly generated early in the rendering pipeline, for example after sprite positions have been updated and / or sprite selections.

[0067] The motion vectors can then be transmitted to the client, and used by the client to estimate the next image frame potentially before the image frame has even been completed at the server, so that optionally an estimate of the next image frame can be displayed with zero downstream latency.

[0068] Once a representative version of the image has been generated by the rendering pipeline at the server, new conventional image based motion vectors may be produced and compared with the earlier motion vectors, with only those differing by a threshold amount being sent to enable an updated estimate of the image at the client. A non-limiting example of a format for such a partial motion vector map is as a GIF image (graphical interchange format), where regions of the image that do not contain motion vectors can be efficiently compressed. Other efficient data formats for sparse arrays can also be considered.

[0069] The updated estimated image can be generated and displayed more quickly than a conventionally transmitted image or motion vector based image, as described previously.

[0070] Finally, based on a server-side construction of the updated estimated image, error correction data can be transmitted as well. This may be transmitted as a single set of data, or separately as first (high priority) and second (low priority) data. High priority data may relate to elements of the image that cannot be easily synthesised from the prior image and motion vectors such as new areas of a scene revealed by panning that comprise for example texture elements not present in the prior image. High priority data may alternatively or in addition relate to salientareas of the image for example based on gaze tracking, the location of the player’s own character, the central third of the image, or the like. Lower priority data may correct inaccuracies in these or other parts of the image.

[0071] In this way, the image may undergo a final second update to reach its best quality, and do so earlier than a conventionally transmitted image or motion vector based image, as described previously.

[0072] Notably, receiving the error correction data means that an accurate version of the image is available to the client when the motion vectors representing the next frame are received. Hence even if a second update is not generated based on the error correction data, the fidelity of the images upon which subsequent images are based does not degrade over time.

[0073] Typically, meanwhile, a scene change will result in an image being transmitted in a traditional manner, since it will have little or no image elements in common with the preceding image, but because it is a jump cut, the change in responsiveness is not so noticeable. Also, scene changes or jump cuts tend to occur during non-interactive cut scenes or during dialogue, where responsiveness to user input is less important.Variant Embodiments

[0074] The techniques herein disclose transmitting motion vectors based on the viewpoint and / or geometry (e.g. only), optionally after z-culling, so that an approximation of the new image can be generate at the client potentially before the new image has been completed at the server, and typically before the completed image could normally be received by the client.

[0075] Optionally, this motion vector data can be updated once the image is sufficiently through the rendering pipeline to be representative of the complete image, and the motion data that is different (i.e. due to non-camera or geometry changes) can be sent to generate a better approximation at the client.

[0076] Whether one or two sets of motion vector data get sent, subsequently the approximate image may then be improved using error correction data as described elsewhere herein.

[0077] However, the above approach need not be limited to just two reviews of the motion data (or the two reviews described previously).

[0078] In particular, by stage 350 of the rendering pipeline both textures and static lighting will have been applied, and so a number of the most prominent sources of change may have beenestablished - in particular shadows, and new image content (e.g. new texture data the enters view during a panning action).

[0079] Consequently, in a variant embodiment of the description, the approach of generating image-based motion vectors may be performed at this earlier stage, and again vectors that differ by more than a predetermined amount from those generated even earlier may be transmitted to the client device at this point.

[0080] Meanwhile optionally also ‘error correction data’ related to new content may also be transmitted at this stage, so that for example there is no missing strip of image data when the approximate new image is displayed. Furthermore, as this data will approximate the image one completed (e.g. with further post-processing effects etc), then the final additional error correction data for this region will be smaller than if nothing had been sent previously.

[0081] Hence in a variant embodiment, image based motion vectors and optionally error correction data (in particular for new image content), can be transmitted ahead of completion of the new image once there is image data available in the pipeline (e.g. after one or more of the texturing stage 340, static lighting stage 350, or dynamic lighting stage 360; the choice of stage being a trade-off between overall delay and fidelity of image data for motion vectors and optional early correction).

[0082] One aspect of a video game image that may not move, but might change, is the so-called user interface overlay; the parts of the image that are typically not part of the rendered virtual environment but are overlaid on top, often as a final step. This may provide, for example, an indication of health, a countdown timer, icons indicating equipped weapons or spells, or the like. Typically it does not move, but next or numbers can change occasionally. Hence optionally this UI overlay can be provided as a separate graphical stream independent of the rest of the image, and may be provided whenever it is generated (possibly in a separate process to the rendering pipeline and hence also earlier that would otherwise be these case). In any case, optionally image based motion vectors may be skipped for the regions of the image that comprise or will be overlaid by such UI overlay elements or at least some parts thereof, since they will not move and any specific changes can be conveyed by error correction data.

[0083] It will be appreciated that the techniques herein provide the client device with several output options: the client could wait until all the image data has been received (all motion vectors and correction data), and generate a final, full quality image. As noted elsewhere, the techniques herein typically mean this can be achieved with less delay than with conventional techniques.

[0084] Alternatively, the client device may only wait until the updated image based motion vector data has been received, and optionally any error correction data, generated once image data was available in the rendering pipeline but before the image was complete - if this optional interim data was generated. This would allow the client device to generate an interim, approximate image, that would include for example new texture data and shadows (depending on the stage at which the data was generated), and potentially display this with much less delay that with conventional techniques. However, for the brief period during which this image is displayed before a full quality image is displayed, there might be some impact on image quality, and possibly the introduction of slight flicker as two versions of the same image get displayed.

[0085] Alternatively, the client device may only wait until the first geometry and / or camera viewpoint based motion vector data has been received. This would allow the client device to generate an early, very approximate image, with little or no effective delay compared to local image generation. However, for the brief period during which this image is displayed before either an updated approximation or a full quality image is displayed, there is likely to be some impact on image quality, and possibly the introduction of slight flicker as two or three versions of the same image get displayed.

[0086] Hence in a variant embodiment of the present description, which of these three approaches is used may be selected as a function of estimated bandwidth and / or ping between the server and the client.

[0087] Hence if there is good bandwidth and / or ping (for example within a first quality of service criterion), then the client may wait for all the data to be received and only output the full quality image, with relatively little overall delay due to relatively little downstream delay being present. Optionally in this case the server may also modify its transmissions to transmit combined data later in the rendering process (for example sending combined first and second motion vector data and error correction data close to or when the rendering of the image is complete).

[0088] Meanwhile if there is poorer bandwidth and / or ping (for example below the first quality of service criterion but still above a second quality of service criterion), then the client may wait for the second motion vector data and any optional error correction data, so that a reasonable image can still be provided in a timely manner pending a better version when the full data is received. Again in this case the server may modify its transmissions, for example to send the first and replacement second motion data together.

[0089] Then, if the bandwidth and / or ping falls below even the second quality of service criterion, then the client may respond to the first motion vector data, when received, to generate an approximate image in a timely manner pending a better version when the optional updated data, and / or the full data, is received.

[0090] Hence the amount of data and optionally it timing according to the techniques herein may be adjusted in response to the needs of the client device with respect to the quality of downstream transmission. As such it is similar by analogy with image compression techniques where a highly compressed base image is transmitted, and where bandwidth allows enhancement layers are also transmitted to improve the image, so that image quality varies with network quality, but the frame rate is maintained. Here, there is a temporal version of the approach where, if needed, data producing ever more approximate versions of the image is sent ever earlier in the rendering cycle so that something can still be displayed in a timely and responsive manner at the client as the network conditions get worse.

[0091] The techniques described herein may be provided as part of a software developer kit, to enable developers to bake such a process into their game. Alternatively or in addition such a software developer kit may include an API, allowing an external process such as an OS (or helper app thereof) of the hosting server to control when / where in the rendering pipeline to generate the motion and / or error correction data to transmit to a client. Alternatively or in addition, the OS or helper app may be equipped to monitor or intervene in the rendering pipeline, for example to generate motion vectors based on the camera and / or geometry data once it is generated, and to generate image based motion vectors and error correction data, as needed, without modification to the game. In other words, particularly for legacy games, the game operates as normal and the OS or helper app monitors the rendering pipeline and generates data as described herein without specific modification of the legacy game code. The legacy game may operate as if it was running on a local machine (or if it was written to have a cloud hosting mode, then as if running conventionally as such), and simply output image frames as normal, whilst the server outputs motion vectors and optionally updated vectors, and error correction data at various times as described elsewhere herein to be used by a corresponding client device to generate images earlier than if transmitted conventionally.Summary

[0092] The use of motion vectors to construct new images is of particular benefit for videogames, where basically the same or similar image elements move from one image frame to the next; the relatively simple graphical textures and lighting, even for realistic looking games,reduces the level of error that occurs when constructing a new image from an old image when both images are using essentially the same textures and similar lighting. By contrast in real-world video, there tends to be more image-to-image variability in surface textures and lighting that results in greater amounts of error correction data.

[0093] As a result the techniques herein are particularly beneficial for streaming videogames (or more generally, rendered content), and the use of motion vectors and optionally error correction data based on the output of early stages of the rendering pipeline rather than or as well as at the end of the pipeline mean that representative images can be output by a client device in a cloud streaming system, with little or no downstream delay compared to local image generation, thereby making the images displayed to the player more responsive to their inputs than could otherwise be the case..

[0094] Accordingly, and referring now to Figure 5, in a summary embodiment a method of streaming for rendered images by a server, wherein the rendered images are generated in a multistage rendering pipeline, the method comprising the following steps.

[0095] In a first step s510, for a next image following a preceding image, generating first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image, as described elsewhere herein.

[0096] In a second step s520, transmitting the first motion vectors to a client device before rendering of the next image is complete, thereby enabling the client device to construct an approximation of the next image using the preceding image and the first motion vectors, as described elsewhere herein.

[0097] It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to that:the method may further comprise generating second motion vectors for the next image based upon current image data of the next image, comparing the second motion vectors to the first motion vectors, and transmitting to the client device those second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount, thereby enabling the client device to construct an approximation of the next image using the preceding image and a combination of the first motion vectors and the transmitted second motion vectors, as described elsewhere herein;the method may further comprise the steps of generating an approximation of the next image based upon whatever motion vector data has been transmitted to the client device for the next image; calculating differences between the approximation of the next image and current image data of the next image, to generate error correction data for the next image; and transmitting the error correction data for the next image to the client device, thereby enabling the client device to correct its approximation of the next image according to current image data of the next image, as described elsewhere herein; the current image data of the next image is obtained after one selected from the list consisting of a texturing stage of the rendering pipeline, a static lighting stage of the rendering pipeline, a dynamic lighting stage of the rendering pipeline, an atmospherics stage of the rendering pipeline, a post-processing stage of the rendering pipeline, and the end of the rendering pipeline, as described elsewhere herein;one or more predetermined sub-regions of the next image are excluded from the method, as described elsewhere herein;in this instance, optionally the one or more predetermined sub-regions of the next image are transmitted according to one selected from the list consisting of when they are generated, when they are integrated into current image data of the next image, and when rendering of the next image, including the or each predetermined sub-region, is complete, as described elsewhere herein; andmotion vectors and any error correction data are transmitted to the client at a timing responsive to the network conditions between the server and the client, as described elsewhere herein.

[0098] Referring now to Figure 6, in a summary embodiment of the present description, a method of displaying streamed rendered images for a client, wherein the rendered images are generated in a multi-stage rendering pipeline, the method comprises the following steps:

[0099] In a first step s610, for a next image following a preceding image, receiving from a server first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image, the first motion vectors having been transmitted before the next image is completed at the server, as described elsewhere herein; and

[0100] In a second step s620, constructing an approximation of the next image using the preceding image and the first motion vectors, as described elsewhere herein.

[0101] It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described andclaimed herein are considered within the scope of the present invention, including but not limited to that:the method may further comprise the steps of receiving from the server second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount; and constructing an approximation of the next image using the preceding image, the second motion vectors, and the first motion vectors not corresponding to received second motion vectors, as described elsewhere herein; andthe method may further comprise the steps of receiving error correction data for the next image from the server, the error correction data being based on differences between an approximation of the next image based upon the preceding image and whatever motion vector data has been transmitted to the client device for the next image; and correcting the client device’s approximation of the next image using the error correction data, as described elsewhere herein.

[0102] It will be appreciated that the above methods may be carried out on hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware.

[0103] Thus the required adaptation to existing parts of an equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0104] Accordingly, and referring again to Figures 1 and 2, in a summary embodiment of the present description, a server (10, 200) for streaming rendered images, wherein the rendered images are generated in a multi-stage rendering pipeline, comprises the following.

[0105] A motion vector processor (e.g. CPU 20 and / or GPU 30) configured (for example by suitable software instruction) to generate, for a next image following a preceding image, first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image, as described elsewhere herein.

[0106] A transmitter (e.g. data port 60 and CPU 20) configured (for example by suitable software instruction) to transmit the first motion vectors to a client device before rendering of the next image is complete, thereby enabling the client device to construct an approximation of the next image using the preceding image and the first motion vectors.

[0107] Instances of this summary embodiment implementing the methods and techniques described herein (for example by use of suitable software instruction) are envisaged within the scope of the application, including but not limited to that:the motion vector processor is configured to generate second motion vectors for the next image based upon current image data of the next image, a comparison processor (e.g. CPU 20) is configured (for example by suitable software instruction) to compare the second motion vectors to the first motion vectors, and the transmitter is configured to transmit to the client device those second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount, thereby enabling the client device to construct an approximation of the next image using the preceding image and a combination of the first motion vectors and the transmitted second motion vectors, as described elsewhere herein.

[0108] Similarly, and referring again to Figures 1 and 2, in a summary embodiment of the present description, a client device (10, 140) for displaying streamed rendered images, wherein the rendered images are generated by a server in a multi-stage rendering pipeline, comprises the following.

[0109] A receiver (e.g. data port 60 and CPU 20) configured (for example by suitable software instruction) to receive from the server first motion vectors for a next image following a preceding image, the first motion vectors being based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image, the first motion vectors having been transmitted before the next image is completed at the server, as described elsewhere herein.

[0110] A graphics processor (e.g. CPU 20 and / or GPU 30) configured (for example by suitable software instruction) to construct an approximation of the next image using the preceding image and the first motion vectors (e.g. thereby generating at least an approximate image earlier than would be the case if constructed after the image was complete at the server, and typically before the currently displayed image has been displayed for its full expected period), as described elsewhere herein.[OHl] Instances of this summary embodiment implementing the methods and techniques described herein (for example by use of suitable software instruction) are envisaged within the scope of the application, including but not limited to that:the receiver is configured to receive from the server second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount, and the graphics processor is configured to construct an approximation of the next image using the preceding image, the second motion vectors, and the first motion vectors not corresponding to received second motion vectors, as described elsewhere herein; and the receiver is configured to receive error correction data for the next image from the server, the error correction data being based on differences between an approximation of the next image based upon the preceding image and whatever motion vector data has been transmitted to the client device for the next image, and the graphics processor is configured to correct the client device’s approximation of the next image using the error correction data, as described elsewhere herein.

[0112] It will also be appreciated with reference to Figure 2 that a system for streaming rendered images may be provided, and comprise the server (10, 200) and any supported number of client devices (10,140).

[0113] It will similarly be appreciated that the above embodiments, variant embodiments, and summary embodiments of the description can be combined as appropriate.

[0114] The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

WHAT IS CLAIMED IS:

1. A method comprising:for a next image following a preceding image, generating first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image; andtransmitting the first motion vectors to a client device before rendering of the next image is complete, thereby enabling the client device to construct an approximation of the next image using the preceding image and the first motion vectors.

2. The method of claim 1, further comprising:generating second motion vectors for the next image based upon current image data of the next image;comparing the second motion vectors to the first motion vectors; and transmitting to the client device those second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount, thereby enabling the client device to construct an approximation of the next image using the preceding image and a combination of the first motion vectors and the transmitted second motion vectors.

3. The method of any preceding claim, further comprising:generating an approximation of the next image based upon whatever motion vector data has been transmitted to the client device for the next image;calculating differences between the approximation of the next image and current image data of the next image, to generate error correction data for the next image; and transmitting the error correction data for the next image to the client device, thereby enabling the client device to correct its approximation of the next image according to current image data of the next image.

4. The method of claim 2 or claim 3, in whichthe current image data of the next image is obtained after one selected from the list consisting of:i. a texturing stage of the rendering pipeline;ii. a static lighting stage of the rendering pipeline;iii. a dynamic lighting stage of the rendering pipeline;iv. an atmospherics stage of the rendering pipeline;v. a post-processing stage of the rendering pipeline; andvi. the end of the rendering pipeline.

5. The method of any preceding claim in which:one or more predetermined sub-regions of the next image are excluded from the method.

6. The method of claim 5, in whichthe one or more predetermined sub-regions of the next image are transmitted according to one selected from the list consisting of:i. when they are generated;ii. when they are integrated into current image data of the next image; and iii. when rendering of the next image, including the or each predetermined sub -region, is complete.

7. The method of any preceding claim, in whichmotion vectors and any error correction data are transmitted to the client at a timing responsive to the network conditions between the server and the client.

8. A method comprising:for a next image following a preceding image, receiving, from a server, first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image, the first motion vectors having been transmitted before the next image is completed at the server; andconstructing an approximation of the next image using the preceding image and the first motion vectors.

9. The method of claim 8, further comprising:receiving from the server second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount; andconstructing an approximation of the next image using the preceding image, the second motion vectors, and the first motion vectors not corresponding to received second motion vectors.

10. The method of claim 8 or claim 9, further comprising:receiving error correction data for the next image from the server, the error correction data being based on differences between an approximation of the next image based upon the preceding image and whatever motion vector data has been transmitted to the client device for the next image; andcorrecting the client device’s approximation of the next image using the error correction data.

11. The method of claim 8, claim 9, or claim 10, further comprising: transmitting motion vectors and any error correction data from the server at a timing responsive to the network conditions between the server and the client.

12. A computer program that, when implemented on a processor, performs the method of any preceding claim.

13. A server compri sing :a motion vector processor configured to generate, for a next image following a preceding image, first motion vectors based upon a change in at least one of scene geometry and camera viewpoint between the next image and the preceding image; anda transmitter configured to transmit the first motion vectors to a client device before rendering of the next image is complete, thereby enabling the client device to construct an approximation of the next image using the preceding image and the first motion vectors.

14. The server of claim 12, whereinthe motion vector processor is configured to generate second motion vectors for the next image based upon current image data of the next image;a comparison processor is configured to compare the second motion vectors to the first motion vectors; andthe transmitter is configured to transmit to the client device those second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount, thereby enabling the client device to construct an approximation of the next image using the preceding image and a combination of the first motion vectors and the transmitted second motion vectors.

15. The server of any preceding claim, whereinthe motion vector processor is configured to generate an approximation of the next image based upon whatever motion vector data has been transmitted to the client device for the next image;the comparison process is configured to calculate differences between the approximation of the next image and current image data of the next image, to generate error correction data for the next image; andthe transmitter is configured to transmit the error correction data for the next image to the client device, thereby enabling the client device to correct its approximation of the next image according to current image data of the next image.

16. The server of claim 14 or claim 15, in whicha current image processor is configured to obtain the current image data of the next image after selecting one from the list consisting ofvii. a texturing stage of the rendering pipeline;viii. a static lighting stage of the rendering pipeline;ix. a dynamic lighting stage of the rendering pipeline;x. an atmospherics stage of the rendering pipeline;xi. a post-processing stage of the rendering pipeline; andxii. the end of the rendering pipeline.

17. The server of any preceding claim in which:a sub-region processor is configured to exclude one or more predetermined subregions of the next image.

18. The server of claim 17, in whichthe sub-region processor is configured to transmit the one or more predetermined sub-regions of the next image according to one selected form the list consisting ofiv. when they are generated;v. when they are integrated into current image data of the next image; and vi. when rendering of the next image, including the or each predetermined sub -region, is complete.

19. The server of any preceding claim, in whichthe transmitter is configured to transmit the motion vectors and any error correction data to the client at a timing responsive to the network conditions between the server and the client.

20. A client device comprising:a receiver configured to receive from the server first motion vectors for a next image following a preceding image, the first motion vectors being based upon a change in at least one of scene geometry and camera viewpoint between the next image and the precedingimage, the first motion vectors having been transmitted before the next image is completed at the server; anda graphics processor configured to construct an approximation of the next image using the preceding image and the first motion vectors.

21. The client device of claim 20, in which:the receiver is configured to receive from the server second motion vectors that differ from the corresponding first motion vectors by more than a threshold amount; andthe graphics processor is configured to construct an approximation of the next image using the preceding image, the second motion vectors, and the first motion vectors not corresponding to received second motion vectors.

22. The client device of claim 20 or claim 21, further comprising: the receiver is configured to receive from the server receiving error correction data for the next image, the error correction data being based on differences between an approximation of the next image based upon the preceding image and whatever motion vector data has been transmitted to the client device for the next image; andthe graphics processor configured to correct the client device’s approximation fo the next image using the error correction data.

23. The client device of claim 20, claim 21, or claim 22, further comprising: the receiver is configured to receive motion vectors and any error correction data from the server at a timing responsive to the network conditions between the server and the client.

Images