Screen generation method and apparatus

Abstract

The present application provides a screen generation method and apparatus. The method comprises: acquiring a resource release time interval; on the basis of the resource release time interval, periodically releasing resources in a resource pool, the resources being used for generating a screen; and when a screen generation request is received, using target resources released from the resource pool to generate a target screen requested to be generated by the screen generation request. According to the method, screen stuttering can be prevented or reduced while preventing screen tearing.

Description

A method and apparatus for generating images

[0001] This application claims priority to Chinese Patent Application No. 202411595015.7, filed on November 8, 2024, entitled "A Method and Apparatus for Generating an Image", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a method and apparatus for generating images. Background Technology

[0003] In real-time interactive scenarios, the display's performance is affected by the stability of the generated images. With increasingly abundant computing resources, the image generation rate is accelerating, potentially exceeding the display's refresh rate. This difference in generation speed leads to issues like screen tearing.

[0004] In related technologies, to avoid screen tearing, control signals are used to control screen generation. In this technology, screen generation is only initiated when a control signal is issued. The frequency of these control signals is no higher than the display's refresh rate, ensuring that the screen generation rate does not exceed the display's refresh rate, thus preventing screen tearing.

[0005] However, this technology increases the latency of image generation. For example, when the reception time of the image generation request falls between the transmission times of two control signals, the generation of the requested image cannot begin until the next control signal of those two control signals is transmitted, which increases the latency of image generation. When the reception rate of the image generation request is low, this latency may lead to problems such as image stuttering. Summary of the Invention

[0006] This application provides a screen generation method and apparatus that can avoid screen tearing while avoiding or reducing screen stuttering.

[0007] In a first aspect, a screen generation method is provided, which is applied to a screen generation device. The method includes: obtaining a resource release time interval; periodically releasing resources in a resource pool based on the resource release time interval; wherein the resources are used to generate a screen; and when a screen generation request is received, using the target resources released from the resource pool to generate the target screen requested by the screen generation request.

[0008] The resource release interval reflects the upper limit of the frame generation rate. The smaller the resource release interval, the higher the upper limit of the frame generation rate. The larger the resource release interval, the lower the upper limit of the frame generation rate.

[0009] This method controls the release rate of resources required for screen generation by controlling the resource release time interval, thereby controlling the screen generation rate. This control ensures that the screen generation rate does not exceed the upper limit of the screen generation rate. Thus, even when the screen generation request receiving rate is too high, the screen generation rate will not be too high, thereby avoiding screen tearing caused by the screen generation rate exceeding the refresh rate of the display device.

[0010] Meanwhile, resources released from the resource pool are used for screen generation. If the screen generation request reception rate is lower than the released resources, the released resources are idle and can be used for screen generation at any time. In this case, screen generation can be started immediately whenever a screen generation request is received, thereby reducing screen generation latency and avoiding or reducing screen stuttering.

[0011] In short, the method provided in this application can avoid screen tearing while avoiding or reducing screen stuttering.

[0012] In one possible implementation, the image generation device communicates with a storage space, which stores images generated by the image generation device, and the images in the storage space are read and displayed by the image display device; obtaining the resource release time interval includes: obtaining the resource release time interval based on the interval between two adjacent images read by the image display device from the storage space.

[0013] The resource release interval is the same as the interval between two adjacent frames read from the storage space by the display device. For example, the resource release interval is equal to the interval between two adjacent frames read from the storage space by the display device. The interval between two adjacent frames read from the storage space by the display device represents the refresh rate of the display device.

[0014] In this implementation, the resource release time interval is obtained by reading the interval between two adjacent frames from the storage space by the display device, ensuring consistency between the resource release time interval and the refresh rate of the display device. Thus, by controlling the frame generation rate through the resource release time interval, consistency between the frame generation rate and the refresh rate of the display device can be guaranteed. Therefore, when displaying images using the display device, screen tearing and stuttering can be avoided or reduced.

[0015] In one possible implementation, the storage space is located remotely to the image generation device; the method further includes sending the target image to the storage space via a network. For example, the storage space is located locally on the user's device, while the image generation device is located in the cloud.

[0016] In scenarios where the image is generated remotely, this method can avoid screen tearing and reduce or eliminate screen stuttering.

[0017] In one possible implementation, resources in the resource pool are periodically released based on a resource release time interval, including: obtaining a first duration required for the image generation device to generate an image in history; obtaining a resource release time based on the first duration and a refresh time; wherein the refresh time is the time when the image display device reads the image from the storage space; the resource release time is a second duration earlier than the refresh time; the second duration is greater than or equal to the first duration, and the second duration is less than the duration of the resource release time interval; and periodically releasing resources in the resource pool according to the resource release time interval, starting from the resource release time.

[0018] In this implementation, the resource release time interval is consistent with the time interval between the display device reading two adjacent images from the storage space.

[0019] Resources are released from the resource pool at the resource release time and at subsequent times that differ from the resource release time by an integer number of resource release time intervals. The resource release time is a second time interval preceding the refresh time of the display device. In other words, the release of resources from the resource pool precedes the refresh of the display device by a second time interval. This ensures that the start time of image generation is a second time interval preceding the refresh time. The second time interval is greater than or equal to the first time interval required for the image generation device to generate the image. The fact that the start time of image generation is a second time interval preceding the refresh time guarantees that image generation is completed either during or before the refresh of the display device. Therefore, the generated image can be read and displayed by the display device in a timely manner, ensuring the timeliness of image display.

[0020] In one possible implementation, the screen generation request is triggered by the user; when the screen generation request is received, the target screen requested by the screen generation request is generated using the target resources released from the resource pool, including: if all resources released from the resource pool at the time of receiving the screen generation request are in a non-idle state, and the time of receiving is between two adjacent resource release times, then the time interval between the two adjacent resource release times is reduced.

[0021] Generally speaking, the screen generated by a user-triggered screen generation request (i.e., the target screen) is the screen the user urgently needs to see. For example, in a game scenario, the user issues a game command and hopes to see the corresponding screen to receive timely game feedback. Therefore, when the screen generation request is triggered by the user, it is necessary to ensure the timeliness of the screen display.

[0022] If all resources released from the resource pool are not idle when a screen generation request is received, then it is necessary to wait for resources to be released from the resource pool before the screen can be generated. If the time of receiving the screen generation request falls between two adjacent resource release times, it indicates that there is a time interval between receiving the screen generation request and releasing the resources. This time interval is the screen generation waiting time, which affects the timeliness of screen display. In this case, reducing the time interval between the two adjacent resource release times allows resources to be released from the resource pool earlier. This reduces the screen generation waiting time and improves the timeliness of screen display.

[0023] In one possible implementation, reducing the time interval between two adjacent resource release times includes: when the time difference between the receiving time and the subsequent resource release time of the two adjacent resource release times is greater than or equal to a first duration, updating the subsequent resource release time to the receiving time; wherein, the first duration is the duration required by the image generation device to generate the image in history.

[0024] Updating the subsequent resource release time to the time the screen generation request is received allows resources to be released from the resource pool upon receiving the request, enabling immediate screen generation. However, if the time difference between the screen generation request reception time and the subsequent resource release time is less than duration T1, even if the screen is generated immediately upon receiving the request, generation cannot be completed by the subsequent resource release time. Consequently, the screen cannot be read and displayed at the refresh time corresponding to that release time. Furthermore, updating the resource release time incurs computational overhead. When the time difference between the screen generation request reception time and the subsequent resource release time is less than duration T1, updating the resource release time results in unnecessary computational costs.

[0025] In this implementation, the time difference between the receiving time of the screen generation request and the subsequent resource release time is greater than or equal to the duration T1, and the subsequent resource release time is updated to the receiving time of the screen generation request, thus saving unnecessary computational overhead.

[0026] In one possible implementation, the target image requested by the image generation request is generated using the target resources released from the resource pool, including: rendering and / or compositing the target resources to obtain the target image.

[0027] Rendering and compositing are different stages in the image generation process. The method provided in this application has high flexibility and can control the image generation rate by controlling one or more stages in the image generation process, thereby avoiding or reducing image stuttering while avoiding image tearing.

[0028] Secondly, a screen generation apparatus is provided, comprising: an acquisition module for acquiring a resource release time interval; a release module for periodically releasing resources in a resource pool based on the resource release time interval; wherein the resources are used to generate a screen; and a generation module for generating the target screen requested by the screen generation request by utilizing the target resources released from the resource pool when a screen generation request is received.

[0029] In one possible implementation, the image generation device communicates with the storage space, which stores the images generated by the image generation device, and the images in the storage space are read and displayed by the image display device; the acquisition module is used to: obtain the resource release time interval based on the interval between two adjacent images read from the storage space by the image display device.

[0030] In one possible implementation, the storage space is located at the remote end of the image generation device; the generation module is also used to send the target image to the storage space via a network.

[0031] In one possible implementation, the release module is used to: obtain a first duration required for the image generation device to generate the image in history; obtain a resource release time based on the first duration and the refresh time; wherein the refresh time is the time when the image display device reads the image from the storage space; the resource release time is a second duration earlier than the refresh time; the second duration is greater than or equal to the first duration, and the second duration is less than the duration of the resource release time interval; and periodically release resources in the resource pool according to the resource release time interval, starting from the resource release time.

[0032] In one possible implementation, the screen generation request is triggered by the user; the generation module is used to: if all resources released from the resource pool at the time of receiving the screen generation request are in a non-idle state, and the time of receiving is between two adjacent resource release times, then reduce the time interval between the two adjacent resource release times.

[0033] In one possible implementation, the generation module is used to update the next resource release time to the receiving time when the time difference between the receiving time and the next resource release time of two adjacent resource release times is greater than or equal to a first duration; wherein, the first duration is the duration required by the image generation device to generate the image in history.

[0034] In one possible implementation, the generation module is used to: render and / or composite the target resources to obtain the target image.

[0035] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method provided in the first aspect.

[0036] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method provided in the second aspect.

[0037] Fifthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, execute the method provided in the first aspect.

[0038] The beneficial effects of the second to fifth aspects can be referred to the introduction of the beneficial effects of the first aspect above, and will not be repeated here. Attached Figure Description

[0039] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0040] Figure 2 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0041] Figure 3 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0042] Figure 4 is a flowchart of a screen generation method provided in an embodiment of this application;

[0043] Figure 5 is a schematic diagram of resource release provided in an embodiment of this application;

[0044] Figure 6 is a schematic diagram of resource release timing adjustment provided in an embodiment of this application;

[0045] Figure 7 is a schematic diagram of the structure of a screen generation device provided in an embodiment of this application;

[0046] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0047] Figure 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0048] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0049] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" refers to two or more objects, and "various types" refers to two or more types. Terms such as "first," "second," etc., are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0050] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.

[0051] Image: refers to an image displayed on a display device (such as a screen), which can be used to represent people, objects, light, color, etc. Images can be generated using computer technology. Image generation can refer to rendering and compositing, or it can refer to either rendering or compositing.

[0052] Refresh: refers to the process by which a display device updates the image it is showing. Specifically, refreshing may include the display device reading an image from its storage space and displaying the read image.

[0053] Refresh cycle: Consists of the refresh time and the interval between two adjacent refresh times. The interval between two adjacent refresh times is the reciprocal of the refresh rate. The start time of a refresh cycle is the refresh time, and the cycle length is the interval between two adjacent refresh times.

[0054] Refresh rate: also known as refresh frequency, refers to the number of times the display refreshes per unit of time. Generally speaking, the refresh rate of a display device is fixed.

[0055] Video frame rate: This refers to the number of images contained in a unit of video duration. For example, video frame rate can be expressed as frames per second (FPS), which means the number of images in a 1-second video. Here, an image can be a frame of a video.

[0056] Rendering refers to the process of generating the elements that make up a screen. For example, these elements can be one or more 3D models. Rendering can include performing geometric transformations, projection transformations, perspective transformations, and window clipping on these 3D models, as well as acquiring material and lighting data, to generate the elements that contain these 3D models. Typically, rendering is performed by an application (APP).

[0057] Screen components: These are the images or sub-screens that make up the screen. Typically, screen components include scenes (such as a 3D scene composed of one or more 3D models), control images, etc.

[0058] Compositing: This refers to the process of combining multiple screen elements into a single image. Multiple screen elements can be arranged according to a pre-defined layout to create the final image. Typically, screen compositing is performed by the operating system (OS).

[0059] Resources refer to the hardware, software, and data that support program execution and enable related functions. For example, resources can include processor resources, storage resources, and network resources. Processor resources can include a central processing unit (CPU) and a graphics processing unit (GPU). Storage resources can include memory, video memory, and hard disks. Network resources can include network interface controllers (NICs) and bandwidth. Furthermore, CPUs, GPUs, memory, hard disks, and NICs can be physical hardware or virtualized versions of physical hardware.

[0060] A resource pool refers to a collection or container containing multiple resources. In fields such as cloud computing, a resource pool can refer to infrastructure.

[0061] Infrastructure refers to the facilities that provide computing resources, storage resources, and / or network resources for computing services. Infrastructure may include at least one data center, each data center comprising multiple physical machines (e.g., servers), where each physical machine can provide one or more resources.

[0062] In fields such as gaming, virtual reality (VR), digital humans, cloud phones, and cloud desktops, the timely display of visuals (especially user-triggered visuals) is crucial to the user experience.

[0063] In related technologies, to avoid screen tearing caused by the image generation rate exceeding the display refresh rate, a control signal related to the display refresh rate is used to control the image generation rate. Specifically, when an image generation request is received, image generation does not begin immediately, but waits for a control signal to trigger image generation, and one control signal triggers only the image generation corresponding to one image generation request.

[0064] Generally, image generation requests are not received at a uniform rate; they vary in speed. When the rate at which image generation requests are received exceeds the rate at which control signals are transmitted, this scheme can reduce the image generation rate to a level no greater than the display's refresh rate, thus preventing image tearing. Conversely, when the rate at which image generation requests are received is less than the rate at which control signals are transmitted, this scheme also reduces the image generation rate to a level less than the rate at which image generation requests are received. In this case, reducing the image generation rate is detrimental, affecting the timely display of the image and causing stuttering.

[0065] This application provides a screen generation method. This method periodically releases resources from a resource pool according to resource release time intervals to control the release rate of resources used for screen generation. This reduces the screen generation rate when the screen generation request reception rate is high, and does not affect the screen generation rate when the screen generation request reception rate is low, ensuring timely screen generation. In summary, if the screen generation request reception rate is high, for example, if two or more screen generation requests are received within one resource time interval, the currently released resources are only used to generate the screen requested by one screen generation request, thus ensuring that the screen generation rate is consistent with the resource release time interval, ensuring smooth screen generation and avoiding screen tearing. If the screen generation request reception rate is low, for example, if one screen generation request is received within two or more resource release time intervals, the reception rate of the screen generation request is less than the resource release rate. The released resources wait for the screen generation request. Thus, whenever a screen generation request is received, resources can be used immediately to generate the screen, ensuring timely screen generation and, consequently, timely screen display, avoiding or reducing screen stuttering.

[0066] Next, the image generation method provided in the embodiments of this application will be described in detail.

[0067] Figure 1 illustrates a system architecture 100 that can be used to implement this method. The system architecture includes a screen generation device 110 and a resource pool 120.

[0068] Resource pool 120 contains multiple resources, such as resource 121, resource 122, etc. For example, the resources in resource pool 120 may be purchased or leased by tenants within the infrastructure. Resources in resource pool 120, when released, can be used to generate images. Resources not released in resource pool 120 are not used to generate images. Therefore, the image generation rate can be controlled by controlling the rate at which resources in resource pool 120 are released.

[0069] The image generation device 110 can control the release of resources in the resource pool 120 and use the released resources to generate images. Specifically, the image generation device 110 can periodically release resources from the resource pool 120 according to resource release time intervals. The released resources can wait for use by the image generation device 110. When the image generation device 110 receives an image generation request, it can immediately use the resources released from the resource pool 120 to generate the image requested by that request. Each time, the released resources are used to generate the image requested by one image generation request. In this way, the image generation rate can be controlled to avoid becoming too high when the image generation request reception rate is high. Furthermore, when the image generation request reception rate is low, the image can be generated immediately upon receiving each image generation request, ensuring the timeliness of image generation.

[0070] In some embodiments, as shown in FIG2, the image generation apparatus 110 may include a periodic signal controller, a resource release controller, and a generation module. The periodic signal controller may send resource release signals to the resource release controller. The resource release controller may respond to the resource release signals and release resources in the resource pool 120. Specifically, whenever a resource release signal is received, the resource release controller releases resources from the resource pool 120. That is, the time interval between two adjacent resource release signals sent by the periodic signal controller determines the resource release time interval.

[0071] Referring again to Figure 2, the resources released from resource pool 120 are used by the generation module to generate the screen. Whenever the generation module receives a screen generation request, it can send a resource request to the resource release controller. If there are idle resources among the resources released from resource pool 120, the resource release controller can respond to the resource request and allocate the idle resources to the generation module for its use. If there are no idle resources among the resources released from resource pool 120, the controller waits to release resources from resource pool 120 again before allocating those resources to the generation module. Idle resources refer to resources that have been released from resource pool 120 but have not been used for screen generation. In other words, idle resources are those released from resource pool 120 other than those currently used for screen generation or already used for screen generation. Furthermore, resources released from resource pool 120 that are currently used for screen generation or already used for screen generation can be considered non-idle resources or resources in a non-idle state.

[0072] In some embodiments, resources released from resource pool 120 can be used for rendering and compositing of an image. That is, the generation module can utilize the resources released from resource pool 120 to render and compose an image. For example, the generation module includes an application and an operating system. The application is used for image rendering, and the operating system is used for image compositing. The resources released from resource pool 120 support the operation of the application, enabling image rendering and obtaining the constituent elements of the image. The resources released from resource pool 120 support the operation of the operating system, compositing the constituent elements of the image to obtain the final image.

[0073] In some embodiments, the resources released from resource pool 120 can be used for rendering the image. That is, the generation module can utilize the resources released from resource pool 120 to render the image and obtain its constituent elements. These constituent elements can be used to composite the image. The generation module can be an application. The resources released from resource pool 120 support the operation of the application, enabling the rendering of the image and obtaining its constituent elements.

[0074] In some embodiments, resources released from resource pool 120 can be used for image composition. That is, the generation module can utilize the resources released from resource pool 120 to composite the constituent elements of an image to obtain an image. For example, the constituent elements of the image can be carried in an image generation request. The generation module can be an operating system. The resources released from resource pool 120 support the operation of the operating system to composite the constituent elements of the image into an image.

[0075] In some embodiments, as shown in FIG2, when a certain screen is generated, the resources required to generate the screen can be recycled. Here, resource recycling refers to returning the resources to the resource pool 120 for later release from the resource pool 120.

[0076] In some embodiments, as shown in Figure 1 or Figure 2, the system architecture also includes a user. The aforementioned screen generation request can be issued by the user. For example, in a game scenario, the screen generation request can be a game operation issued by the user to a person or object in the game.

[0077] In some embodiments, as shown in FIG1 or FIG2, the system architecture further includes a storage space 130 and a screen display device 140. The storage space 130 communicates with the screen generation device 110 so that the screen generation device 110 can store generated screens in the storage space 130. The screen display device 140 communicates with the storage space 130 to read screens from the storage space 130. The screen display device 140 can display the screens read from the storage space 130 to the user.

[0078] In some embodiments, the resource release time interval is based on the interval between the time it takes for the display device 140 to read two adjacent frames from the storage space 130. This ensures consistency between the resource release rate and the frame reading rate, and consequently, consistency between the frame generation rate and the frame reading rate.

[0079] In one example of this embodiment, as described above, the resource release time interval is actually the time interval for sending the resource release signal. Therefore, the time interval for the periodic signal controller to send the resource release signal can be obtained based on the interval between two adjacent frames read from the storage space 130 by the display device 140, thus obtaining the resource release time interval. As shown in FIG2, the periodic signal controller can obtain the refresh cycle of the display device 140, the length of which is the interval between two adjacent frames read from the storage space 130 by the display device 140. Then, the periodic signal controller can obtain the time interval for sending the resource release signal based on this interval. In one example, this interval is equal to the duration of the time interval for sending the resource release signal by the periodic signal controller.

[0080] The refresh cycle may also include the refresh time of the display device 140, which may specifically be the time when the image is read from the storage space 130. Based on this refresh time, the start time of the periodic signal controller sending the resource release signal can be obtained, thereby obtaining the start time of releasing resources from the resource pool 120. Specific details will be described in the method embodiments below and will not be repeated here.

[0081] In some embodiments, the system architecture shown in Figure 1 can specifically be a terminal device, such as a mobile phone, personal computer, smart wearable device, vehicle terminal, smart TV, etc. The image generation device 110 can be a computing component in the terminal device, the resource pool 120 can be the hardware resources of the terminal device, the storage space 130 can be a storage component (e.g., a cache corresponding to the display component), and the image display device 140 can be the display component (e.g., a display screen) of the terminal device.

[0082] In some embodiments, the image generation device 110 may be a device, equipment, module, platform, or cluster located remotely at the storage space 130. In this embodiment, as shown in FIG3, the image generation device 110 can send the image to the storage space 130 via a network for storage. Furthermore, the image display device 140 and the storage space 130 may be related components in a terminal device; for example, the image display device 140 is the terminal device's display screen, and the storage space 130 is the cache corresponding to the terminal device's display screen. The terminal device is located locally at the user's location, and the user can send image generation requests to the image generation device 110 via a network.

[0083] The above example illustrates the system architecture provided in the embodiments of this application. Next, taking implementation within this system architecture as an example, the execution flow of the screen generation method provided in the embodiments of this application will be described. This method can be executed by the screen generation device 110 in the system architecture. As shown in Figure 4, the method includes the following steps.

[0084] Step 401: The image generation device 110 obtains the resource release time interval.

[0085] In some embodiments, the resource release time interval can be configured. For example, a user can input a desired video frame rate, and the image generation device 110 obtains the resource release time interval based on that video frame rate. For instance, if the video frame rate is expressed in fps, the resource release time interval can be obtained by dividing the duration of one second by the video frame rate.

[0086] In some embodiments, the image generation device 110 communicates with the storage space 130, which stores images generated by the image generation device 110, and the images in the storage space 130 are read and displayed by the image display device 140. In this embodiment, the resource release time interval can be obtained based on the interval between two adjacent images read by the image display device 140 from the storage space 130. For example, the image generation device 110 can obtain the refresh cycle of the image display device 140, and obtain the resource release time interval based on the cycle length of the refresh cycle (i.e., the interval between two adjacent images read by the image display device 140 from the storage space 130). In one example, the resource release time interval is equal to the cycle length of the refresh cycle.

[0087] Step 402: The image generation device 110 periodically releases resources in the resource pool 120 based on the resource release time interval; wherein the released resources are used to generate images.

[0088] Specifically, after each resource release time interval, the image generation device 110 performs a resource release operation, which refers to the operation of releasing resources from the resource pool 120. Through the resource release operation, resources can be released from the resource pool 120. The resources released from the resource pool 120 in each resource release operation are used to composite an image.

[0089] In some embodiments, referring to Figure 5, a resource release operation can be performed at time t1, releasing resource 121 from resource pool 120. At time t2, which is one resource release time interval from time t1, another resource release operation is performed, releasing resource 122 from resource pool 120. This process continues, with a resource release operation performed after each resource release time interval. The resource release operation is performed only if the resources in resource pool 120 are not empty. If the resources in resource pool 120 are empty, the resource release operation is skipped.

[0090] In some embodiments, the image generation device 110 communicates with the storage space 130, which stores images generated by the image generation device 110, and the images in the storage space 130 are read and displayed by the image display device 140. In step 402, the period for the image generation device 110 to perform the resource release operation is the same as the refresh period of the image display device 140, that is, the time interval is the same and the starting time is the same. In other words, the image generation device 110 performs a resource release operation once every time the image display device 140 reads an image from the storage space 130.

[0091] In some embodiments, the image generation device 110 communicates with the storage space 130, which stores images generated by the image generation device 110, and the images in the storage space 130 are read and displayed by the image display device 140. In step 402, the image generation device 110 can obtain the time T1 required for the image generation device to generate an image in the past. Specifically, the time T1 is the time required for the image generation device 110 to generate one image. For example, the time T1 may be the time required for the image generation device 110 to generate one image in the past using resources released from the resource pool 120.

[0092] The time taken for the image generation device 110 to generate an image is T1. To ensure that the image display device 140 can promptly read and display the image after generation, the time when resources are released from the resource pool 120 can be at least T1 seconds earlier than the refresh time of the image display device 140. The refresh time of the image display device 140 is the time when it reads the image from the storage space 130. In this way, image generation can be completed before the image display device 140 initiates a refresh, allowing the generated image to be displayed promptly without waiting for the next refresh.

[0093] Specifically, after obtaining the duration T1, the image generation device 110 can determine the resource release time based on the duration T1 and the refresh time of the image display device 140. The duration T2 can be obtained based on T1. Duration T2 is greater than or equal to duration T1 and less than the resource release time interval. The refresh time of the display device 140 is advanced by duration T2 to obtain the resource release time. Specifically, the resource release time is advanced by duration T2 compared to the refresh time of the image display device 140. Then, starting from the resource release time, resources in the resource pool are periodically released according to the resource release time interval.

[0094] In some embodiments, as described above, the release of resources in resource pool 120 is triggered by a resource release signal issued by a periodic signal controller. In this embodiment, after the periodic signal controller is activated, it can issue resource release signals in a manner synchronized with the refresh cycle of the display device 140. That is, the time at which the periodic signal controller issues the resource release signal is the same as the refresh time of the display device 140, and the time interval between two adjacent resource release signals is the same as the length of the refresh cycle. In this case, whenever the display device 140 reads a screen from the storage space 130, the resource release controller performs a resource release operation to release resources from resource pool 120.

[0095] As shown in Figure 6, the generation module can generate a screen using resources released from resource pool 120. The generation module can record the time taken to generate the screen using resources released from resource pool 120, obtaining a duration T1. Then, the generation module can feed back the duration T1 to the periodic signal controller via the resource release controller, or the generation module can directly feed back the duration T1 to the periodic signal controller. The periodic signal controller can adjust the timing of the resource release signal based on the duration T1. Specifically, the timing of the resource release signal can be advanced by a duration T2, so that the time of releasing resources from resource pool 120 is advanced by a duration T2 compared to the refresh time of screen display device 140. The duration T2 is greater than or equal to the duration T1, and less than the duration of the resource release time interval.

[0096] Taking a system with a desired video frame rate of 60fps as an example, the resource release interval, or the time interval between two adjacent resource release signals issued by the periodic signal controller, is 16.6ms, meaning a resource release operation is performed every 16.6ms. Assume the duration T1 is 8ms. If the periodic signal controller can issue resource release signals in a way that synchronizes with the refresh cycle of the display device 140, then it will take 8.6ms after the image is generated before it can be displayed. By advancing the timing of the resource release signal issued by the periodic signal controller by 8ms (duration T1 is 8ms), the completion time of image generation becomes the refresh time of the display device 140, thereby reducing the image display waiting time.

[0097] Step 403: When a screen generation request is received, the screen generation device 110 uses the target resources released from the resource pool 120 to generate the target screen requested by the screen generation request. For ease of description, the screen requested by the screen generation request is referred to as the target screen, and the resources used to generate the target screen are referred to as target resources.

[0098] In some embodiments, the screen generation request can be spontaneous; for example, the application can spontaneously generate a screen generation request. For instance, in a game scenario, even if the user does not perform any game actions for a long time, the game application's screen does not display a static image, but rather an animation. In this case, the animation is generated by a screen generation request spontaneously generated by the application.

[0099] In some embodiments, the screen generation request may be user-triggered. For example, a user may issue an operation via an input device (e.g., mouse, keyboard, touchscreen, or microphone), which triggers the screen generation request. For example, the screen generation device 110 may be located remotely to the user, and the screen generation device 110 may receive user-triggered screen generation requests via a network.

[0100] In some embodiments, as described above, each resource released from resource pool 120 is used to generate one frame. For ease of description, each resource released from resource pool 120 can be referred to as a resource, and one resource is used to generate one frame. One resource is released every resource release time interval, so that when a frame generation request is received, at least one resource may have been released from resource pool 120.

[0101] If at least one of the resources is idle, then that idle resource can be used as the target resource, i.e., the target image can be generated using that idle resource. For example, as shown in FIG5, if resource 122 is an idle resource, then the target image can be generated using resource 122. Here, idle resources are those released from the resource pool 120 excluding resources currently being used to generate an image or resources already used to generate an image. Furthermore, resources released from the resource pool 120 that are currently being used to generate an image or resources already used to generate an image can be referred to as non-idle resources or resources in a non-idle state.

[0102] If there are no idle resources among the at least one resource, meaning all resources in the at least one resource are in a non-idle state, then after releasing resources from the resource pool 120 again, the resources released from the resource pool 120 again will be used as target resources to generate the target image. In this way, the image generation rate can be controlled by controlling the release of resources.

[0103] In some embodiments, as shown in FIG5, after the screen is generated, the resources used for the screen generation can be recycled back to the resource pool 120, thereby realizing the recycling of resources.

[0104] In some embodiments, as described above, the screen generation request can be user-triggered. The screen requested by the user-triggered screen generation request (i.e., the target screen) may be something the user urgently needs to see. For example, in a game scenario, the user issues a game operation and hopes to see the changes in the game screen immediately, and to receive timely game feedback, i.e., they want to see the effect of the operation. Therefore, when the screen generation request is user-triggered, it is necessary to ensure the timeliness of the screen display.

[0105] If there are idle resources released from resource pool 120 when a screen generation request is received, the idle resources can be used to generate the target screen without affecting the timeliness of the target screen display.

[0106] If, upon receiving a screen generation request, all resources released from resource pool 120 are in a non-idle state, and the time of receiving the screen generation request falls between two adjacent resource release times, then the time interval between those two adjacent resource release times should be shortened. The fact that all resources released from resource pool 120 are in a non-idle state indicates that resources for generating the target screen will not be released until the next resource release time. The fact that the time of receiving the screen generation request falls between two adjacent resource release times indicates a time interval between receiving the request and releasing resources; this time interval is the screen generation waiting time, which affects the timeliness of the target screen display. In this case, shortening the time interval between the two adjacent resource release times allows resources to be released from resource pool 120 earlier, reducing the waiting time for target screen generation and ensuring the timeliness of the target screen display.

[0107] In one example of this embodiment, reducing the time interval between two adjacent resource release times can specifically be achieved by advancing the release time of the latter resource release time to the time the screen generation request is received, that is, updating the latter resource release time to the time the screen generation request is received. Therefore, when the screen generation request is received, resources for generating the target screen can be released from the resource pool 120, thereby eliminating the waiting time for target screen generation and ensuring the timeliness of target screen display.

[0108] In one example of this example, when the time difference between the receiving time of the screen generation request and the subsequent resource release time of the two adjacent resource release times is greater than or equal to the duration T1, the subsequent resource release time is updated to the receiving time of the screen generation request. The meaning of duration T1 is explained above and will not be repeated here. Furthermore, in this example, duration T1 is less than the resource release time interval. And, the resource release time and the refresh time of the screen display device 140 are consistent. That is, whenever the screen display device 140 reads a screen from the storage space 130, the screen generation device 110 performs a resource release operation.

[0109] Generating the target screen requires a duration T1. If the time difference between the time the screen generation request is received and the subsequent resource release time of the two adjacent resource release times is less than T1, then even if the target screen is generated immediately upon receiving the screen generation request, the generation of the target screen cannot be completed at the subsequent resource release time, and therefore cannot be read and displayed at the refresh time corresponding to the subsequent resource release time. Furthermore, changing or updating the resource release time incurs computational overhead. Therefore, in this example, the subsequent resource release time is only updated to the time the screen generation request is received when the time difference between the time the screen generation request is received and the subsequent resource release time of the two adjacent resource release times is greater than or equal to T1, thus saving unnecessary computational overhead.

[0110] In some embodiments, step 403 may specifically involve rendering related to the image generation request using the target resources to obtain the target image. Specifically, the constituent elements of the target image may be rendered, and these constituent elements can be used to composite the target image to obtain the target image.

[0111] In some embodiments, step 403 may specifically involve using the target resource to perform compositing related to the image generation request to obtain the target image. The image generation request may carry the constituent elements of the target image, and in step 403, the constituent elements of the target image are composited to obtain the target image.

[0112] In some embodiments, step 403 may specifically involve rendering and compositing related to the image generation request using the target resources to obtain the target image. Specifically, the constituent elements of the target image may be rendered, and then the constituent elements of the target image may be composited to obtain the target image.

[0113] After obtaining the target image, the image generation device 110 can store the target image in the storage space 130. The image display device 140 can read the target image from the storage space 130 and display the target image.

[0114] In some embodiments, the storage space 130 is located at a remote end of the image generation device 110. After obtaining the target image, the image generation device 110 can send the target image to the storage space 130 via a network. For example, the image generation device 110 can encode the target image into video frames to send the target image to the storage space 130 via a video stream. For example, the target image can be sent to the storage space 130 via network streaming. After arriving at the storage space 130, the target image can be stored in the storage space 130.

[0115] In summary, the screen generation method provided in this application embodiment can ensure the stability of screen generation and avoid screen tearing caused by the screen generation rate being greater than the refresh rate of the display device, while also ensuring the timeliness of screen generation and avoiding screen stuttering.

[0116] Based on the above description, this application provides a screen generation apparatus 700. As shown in FIG7, the apparatus 700 includes:

[0117] Module 710 is used to obtain the resource release time interval;

[0118] The release module 720 is used to periodically release resources in the resource pool based on the resource release time interval; wherein the resources are used to generate images;

[0119] The generation module 730 is used to generate the target image requested by the image generation request by utilizing the target resources released from the resource pool when an image generation request is received.

[0120] In some embodiments, the device 700 communicates with a storage space, the storage space being used to store images generated by the device 700, and the images in the storage space being read and displayed by a display device; the acquisition module 710 is used to: obtain the resource release time interval based on the interval between two adjacent images read from the storage space by the display device.

[0121] In one example of this embodiment, the storage space is located at a remote end of the device 700; the generation module 730 is further configured to: send the target image to the storage space via a network.

[0122] In some embodiments, the release module 720 is configured to: obtain a first duration required for the image generation device to generate an image in history; obtain a resource release time based on the first duration and a refresh time; wherein the refresh time is the time when the image display device reads the image from the storage space; the resource release time is a second duration earlier than the refresh time; the second duration is greater than or equal to the first duration, and the second duration is less than the duration of the resource release time interval; and periodically release resources in the resource pool according to the resource release time interval, starting from the resource release time.

[0123] In some embodiments, the screen generation request is triggered by the user; the generation module 730 is configured to: if all resources released from the resource pool at the time of receiving the screen generation request are in a non-idle state, and the time of receiving is between two adjacent resource release times, then reduce the time interval between the two adjacent resource release times.

[0124] In one example of this embodiment, the generation module 730 is configured to: update the latter resource release time to the receiving time when the time difference between the receiving time and the latter resource release time of the two adjacent resource release times is greater than or equal to a first duration; wherein, the first duration is the duration required by the image generation device to generate the image in history.

[0125] In some embodiments, the generation module 730 is used to: render and / or composite the target resource to obtain the target image.

[0126] The acquisition module 710, release module 720, and generation module 730 can all be implemented in software or in hardware. For example, the implementation of the acquisition module 710 will be described below. Similarly, the implementation of the release module 720 and generation module 730 can refer to the implementation of the acquisition module 710.

[0127] As an example of a software functional unit, module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0128] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0129] As an example of a hardware functional unit, the acquisition module 710 may include at least one computing device, such as a server. Alternatively, the acquisition module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0130] The multiple computing devices included in the acquisition module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0131] It should be noted that, in other embodiments, the acquisition module 710 can be used to execute any step in the method shown in FIG4, the release module 720 can be used to execute any step in the method shown in FIG4, and the generation module 730 can be used to execute any step in the method shown in FIG4. The steps implemented by the acquisition module 710, the release module 720, and the generation module 730 can be specified as needed. By implementing different steps in the method shown in FIG4 through the acquisition module 710, the release module 720, and the generation module 730, all functions of the device 700 can be realized.

[0132] This application also provides a computing device 800. As shown in FIG8, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other via the bus 802. The computing device 800 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0133] Bus 802 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 802 can include pathways for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, communication interface 808).

[0134] Processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0135] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0136] The memory 806 stores executable program code, which the processor 804 executes to implement the functions of the aforementioned acquisition module 710, release module 720, and generation module 730, thereby realizing the method shown in FIG4. That is, the memory 806 stores instructions for executing the method shown in FIG4.

[0137] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0138] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0139] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method shown in Figure 4.

[0140] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG4. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG4.

[0141] It should be noted that the memory 806 in different computing devices 800 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the device 700. That is, the instructions stored in the memory 806 of different computing devices 800 can implement the functions of one or more modules among the acquisition module 710, release module 720, and generation module 730.

[0142] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 806 in computing device 800A stores instructions for executing the functions of the acquisition module 710. Simultaneously, the memory 806 in computing device 800B stores instructions for executing the functions of the release module 720 and the generation module 730.

[0143] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0144] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device clusters shown in Figures 9 and 10. The difference is that the memory 806 of one or more computing devices 800 in this computing device cluster can store the same instructions for executing the method shown in Figure 4.

[0145] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG4. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG4.

[0146] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method shown in FIG4.

[0147] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a host migration device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method shown in FIG4.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating an image, characterized in that, The method is applied to an image generation device, and the method includes: Get the resource release time interval; Based on the resource release time interval, resources in the resource pool are periodically released; wherein, the resources are used to generate images; When a screen generation request is received, the target screen requested by the screen generation request is generated using the target resources released from the resource pool.

2. The method according to claim 1, characterized in that, The image generation device communicates with the storage space, which is used to store the images generated by the image generation device, and the images in the storage space are read and displayed by the image display device. The resource release time interval includes: The resource release time interval is obtained by reading the interval between two adjacent frames from the storage space based on the display device.

3. The method according to claim 2, characterized in that, The storage space is located at the remote end of the image generation device; the method further includes: sending the target image to the storage space via a network.

4. The method according to claim 2 or 3, characterized in that, The method of periodically releasing resources from the resource pool based on the resource release time interval includes: Obtain the first duration required for the image generation device to generate the image in history; Based on the first duration and the refresh time, the resource release time is obtained; wherein, the refresh time is the time when the screen display device reads the screen from the storage space; the resource release time is a second duration earlier than the refresh time; the second duration is greater than or equal to the first duration, and the second duration is less than the duration of the resource release time interval; Starting from the resource release time, resources in the resource pool are periodically released according to the resource release time interval.

5. The method according to any one of claims 1-4, characterized in that, The image generation request is triggered by the user; The step of generating the target image requested by the image generation request by utilizing the target resources released from the resource pool when a screen generation request is received includes: If all resources released from the resource pool at the time the image generation request is received are in a non-idle state, and the receiving time is between two adjacent resource release times, then the time interval between the two adjacent resource release times is reduced.

6. The method according to claim 5, characterized in that, The reduction of the time interval between the two adjacent resource release times includes: When the time difference between the receiving time and the subsequent resource release time of the two adjacent resource release times is greater than or equal to the first duration, the subsequent resource release time is updated to the receiving time. The first duration is the duration required for the image generation device to generate the image in the past.

7. The method according to any one of claims 1-6, characterized in that, The step of generating the target image requested by the image generation request using the target resources released from the resource pool includes: rendering and / or compositing the target resources to obtain the target image.

8. An image generation device, characterized in that, The device includes: The acquisition module is used to obtain the resource release time interval; A release module is used to periodically release resources in the resource pool based on the resource release time interval; wherein the resources are used to generate images; The generation module is used to generate the target image requested by the image generation request by utilizing the target resources released from the resource pool when an image generation request is received.

9. The apparatus according to claim 8, characterized in that, The image generation device communicates with a storage space, which stores the images generated by the image generation device, and the images in the storage space are read and displayed by the image display device; the acquisition module is used for: The resource release time interval is obtained by reading the interval between two adjacent frames from the storage space based on the display device.

10. The apparatus according to claim 9, characterized in that, The storage space is located at the far end of the image generation device; the generation module is also used to send the target image to the storage space via a network.

11. The apparatus according to claim 9 or 10, characterized in that, The release module is used for: Obtain the first duration required for the image generation device to generate the image in history; Based on the first duration and the refresh time, the resource release time is obtained; wherein, the refresh time is the time when the screen display device reads the screen from the storage space; the resource release time is a second duration earlier than the refresh time; the second duration is greater than or equal to the first duration, and the second duration is less than the duration of the resource release time interval; Starting from the resource release time, resources in the resource pool are periodically released according to the resource release time interval.

12. The apparatus according to any one of claims 8-11, characterized in that, The image generation request is triggered by the user; The generation module is used for: If all resources released from the resource pool at the time the image generation request is received are in a non-idle state, and the receiving time is between two adjacent resource release times, then the time interval between the two adjacent resource release times is reduced.

13. The apparatus according to claim 12, characterized in that, The generation module is used for: When the time difference between the receiving time and the subsequent resource release time of the two adjacent resource release times is greater than or equal to the first duration, the subsequent resource release time is updated to the receiving time. The first duration is the duration required for the image generation device to generate the image in the past.

14. The apparatus according to any one of claims 8-13, characterized in that, The generation module is used to: render and / or composite the target resources to obtain the target image.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-7.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by a cluster of computer devices, the cluster of computer devices performs the method as described in any one of claims 1-7.

Images