Frame generation method, apparatus, chip, server and program product
By utilizing motion information from multiple historical rendering frames and the processing units of the graphics processor to generate intermediate frames and repairing pixel-missing areas, the problem of low frame generation efficiency in existing technologies is solved, achieving efficient, realistic, and smooth frame generation.
Patent Information
- Application Number
- PCT/CN2026/093809
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-06
- Filing Date
- 2026-04-29
- Publication Date
- 2026-11-12
AI Technical Summary
Existing technologies have high computational complexity during frame generation, resulting in low efficiency. Furthermore, existing extrapolation frame technologies rely on optical flow calculations and have limited rendering pipelines, making it impossible to fully utilize computing resources.
By using motion information from multiple historical rendering frames to predict intermediate frames, and using the floating-point and matrix operation units of the graphics processor to generate intermediate frames, combined with the first historical rendering frame to repair pixel missing areas, parallel processing is achieved to improve efficiency.
It reduces the computational complexity of frame generation, improves the efficiency of frame generation and the realism and smoothness of images, and reduces task waiting time.
Smart Images

Figure CN2026093809_12112026_PF_FP_ABST
Abstract
Description
Frame generation methods, devices, chips, servers, and software products
[0001] This application claims priority to Chinese Patent Application No. 202510575299.1, filed with the State Intellectual Property Office of China on May 6, 2025, entitled "Frame Generation Method, Apparatus, Chip, Server and Program Product", the entire contents of which are incorporated herein by reference. Technical Field
[0002] The embodiments of this application relate to the field of computer technology, and more specifically, to the field of graphics rendering technology. The embodiments of this application relate to frame generation methods, apparatus, chips, servers, and computer program products. Background Technology
[0003] Artificial intelligence (AI)-assisted rendering is a cutting-edge technology that combines artificial intelligence algorithms with computer graphics, aiming to accelerate or optimize the generation process of multi-frame images through AI models. AI-assisted rendering can use AI models, such as deep learning models, to replace some computationally intensive steps in the traditional rendering pipeline, achieving a balance between efficiency and quality. AI-assisted rendering can be applied to various scenarios, such as video processing, animation production, virtual reality, and games.
[0004] To improve the efficiency and quality of image generation and provide a smoother user experience, composite frames can be inserted into the rendered video sequence to increase the frame rate. For example, a new frame can be inserted between two consecutive rendered frames. In video applications, frame interpolation refers to generating an intermediate video frame based on the information of the preceding and following frames; in game applications, frame interpolation refers to generating the image information corresponding to the third frame based on the image information of the first two frames. Summary of the Invention
[0005] Embodiments of this application provide a frame generation scheme. According to this scheme, corresponding intermediate frames can be generated using multiple historical rendering frames and time parameters. The generation of these intermediate frames does not depend on other information, thereby reducing computational complexity and improving frame generation efficiency. Thus, by generating future intermediate frames (e.g., predicting the next frame), the number of frames to be rendered can be reduced, saving rendering computational power. Furthermore, after generating intermediate frames, the first historical rendering frame can be used as a guide to finely repair pixel loss areas or other changes in the intermediate frames, further enhancing the realism and smoothness of the image. In other words, the first historical rendering frame provides rich contextual information, enabling a better understanding of the image's structure and texture when repairing intermediate frames, thereby generating more natural and realistic target frames.
[0006] According to a first aspect of this application, a rendering method is provided, which includes generating an intermediate frame corresponding to a time parameter based on multiple historical rendering frames and a time parameter. The method further includes determining a corresponding target frame based on a first historical rendering frame by repairing pixel-missing regions in the intermediate frame, the target frame being located after the multiple historical rendering frames.
[0007] In some embodiments of the first aspect, the intermediate frame includes a first intermediate frame and a second intermediate frame. The method further includes: determining a corresponding first target frame by repairing pixel-missing regions of the first intermediate frame while generating the second intermediate frame. By processing the intermediate frame generation task and the repair task in parallel, computing resources can be fully utilized, task waiting time can be reduced, and thus the generation efficiency of multiple target frames can be significantly improved.
[0008] In some embodiments of the first aspect, generating an intermediate frame corresponding to a time parameter based on multiple historical rendering frames and time parameters includes: generating an intermediate frame corresponding to a time parameter via a floating-point arithmetic unit of a graphics processor (GPU). By utilizing the GPU's floating-point arithmetic unit for intermediate frame generation, the high-performance computing capabilities of the GPU can be fully utilized to achieve high-precision and high-efficiency intermediate frame generation. Since intermediate frame generation requires high-precision floating-point operations to ensure accuracy, and the floating-point arithmetic unit can handle high-precision floating-point operations, image artifacts and errors caused by insufficient precision can be avoided. Furthermore, during the intermediate frame generation process, the high-precision calculations of the floating-point arithmetic unit ensure that the generated intermediate frame maintains consistency with multiple historical rendering frames in terms of color, contrast, etc., thereby improving the realism and image quality of the generated intermediate frame.
[0009] In some embodiments of the first aspect, determining the corresponding target frame by repairing the pixel missing regions of intermediate frames based on a first historical rendering frame includes: receiving intermediate frames sent by a floating-point arithmetic unit via a matrix arithmetic unit of a graphics processor; and determining the corresponding target frame by repairing the pixel missing regions of intermediate frames based on the first historical rendering frame via a matrix arithmetic unit. Since the matrix arithmetic unit can perform calculations on multiple pixels simultaneously during the repair of pixel missing regions of intermediate frames, the repair speed of intermediate frames is greatly improved.
[0010] In some embodiments of the first aspect, the plurality of historical rendering frames includes a second historical rendering frame, the rendering time of the second historical rendering frame being later than the rendering time of other historical rendering frames in the plurality of historical rendering frames.
[0011] In some embodiments of the first aspect, the first historical rendering frame is one or more of a plurality of historical rendering frames, and the first historical rendering frame is different from the second historical rendering frame; or the rendering time of the first historical rendering frame is earlier than the rendering time of each of the plurality of historical rendering frames; or there are a plurality of first historical rendering frames, some of which belong to the plurality of historical rendering frames, and some of which are different from the plurality of historical rendering frames. That is to say, the selection method of the first historical rendering frame is not fixed, and the number of first historical rendering frames is not fixed, thereby providing a relatively flexible selection range of the first historical rendering frame.
[0012] In some embodiments of the first aspect, the time parameter is used to indicate one or more times between the rendering time of the second historical rendering frame and the rendering time of the third rendering frame, the third rendering frame being the next rendering frame to be generated according to a preset frame rate. In this way, by limiting the range of the time parameter, the final generated target frame can be located between the second and third historical rendering frames, thereby achieving time parameter-driven frame rate expansion. That is, dynamic expansion of the rendering frame rate and a significant improvement in visual smoothness can be achieved. Furthermore, when there are multiple time parameters, multiple target frames can be generated, resulting in a larger sequence frame rate and providing a better visual experience for the user.
[0013] In some embodiments of the first aspect, generating an intermediate frame corresponding to the time parameters based on multiple historical rendering frames and time parameters includes: determining the motion trajectory of a target object based on multiple historical rendering frames, wherein the target object is located in a target scene, and the multiple historical rendering frames are obtained by rendering the target scene; generating a corresponding intermediate frame based on the motion trajectory and a second historical rendering frame, wherein the rendering time of the second historical rendering frame is later than the rendering time of other historical rendering frames in the multiple historical rendering frames. In this way, the corresponding intermediate frame can be generated using the motion information contained in multiple historical rendering frames, without needing to recalculate the rendering state of the entire scene or relying on other information, thereby improving the efficiency of frame generation. Furthermore, the motion trajectory provides a clear reference standard for intermediate frame generation, making the generated intermediate frames more accurate and more in line with the physical laws of the real world.
[0014] In some embodiments of the first aspect, generating a corresponding intermediate frame based on a motion trajectory and a second historical rendering frame includes: determining a corresponding motion vector based on the motion trajectory and a time parameter, where the time parameter characterizes the generation time of the intermediate frame; determining the mapping positions corresponding to multiple pixels in the second historical rendering frame based on the motion vector; and generating the corresponding intermediate frame based on the mapping positions corresponding to the multiple pixels. By calculating the motion vector using the motion trajectory and time parameter, the position, direction, and velocity of the target object at the intermediate frame time can be accurately predicted, thereby avoiding ghosting or afterimages caused by simple interpolation. Furthermore, using the second historical rendering frame as the generation reference for the intermediate frame only requires transforming the pixels in the second historical rendering frame to determine the corresponding intermediate frame, thereby improving generation efficiency and saving generation costs. Additionally, since each pixel in the intermediate frame is a precise mapping from the second historical rendering frame, clear textures and details are preserved, resulting in higher quality and better effects for the generated intermediate frame.
[0015] In some embodiments of the first aspect, the plurality of pixels includes a first pixel and a second pixel. Generating a corresponding intermediate frame based on the projection positions corresponding to the plurality of pixels includes: in response to the first pixel and the second pixel having the same mapping position, mapping the first pixel to a corresponding target mapping position, wherein the first depth corresponding to the first pixel is less than the second depth corresponding to the second pixel. In this way, during the mapping process, depth comparison ensures that the mapping position of each pixel is unique and correct, avoiding visual instability and logical errors caused by depth conflicts, and making the motion transition of the intermediate frame more natural.
[0016] In some embodiments of the first aspect, the first historical rendering frame contains the target object corresponding to the pixel-missing region.
[0017] In some embodiments of the first aspect, the method further includes: determining morphological change information of corresponding shadow areas based on multiple consecutive historical rendering frames; and determining a corresponding target frame by adjusting the shadow areas of the intermediate frames based on the morphological change information. By analyzing the shadow morphological changes of historical rendering frames, the shadow areas of intermediate frames can be dynamically adjusted, allowing shadows to transition naturally with the movement of scene objects or changes in lighting, which is more in line with actual physical laws, thereby improving the smoothness, realism, and stability of the image.
[0018] According to a second aspect of this application, a frame generation method is provided. This method includes generating a first intermediate frame corresponding to a first time parameter based on multiple historical rendering frames and a first time parameter. The method further includes, while generating a second intermediate frame, determining a corresponding first target frame based on the first historical rendering frame by repairing pixel-missing regions in the first intermediate frame. The second intermediate frame corresponds to a second time parameter, and the first and second target frames are located after the multiple historical rendering frames. This approach enables parallel processing of the intermediate frame generation and repair tasks, fully utilizing computational resources, reducing task waiting time, and thus significantly improving the generation efficiency of multiple target frames.
[0019] In some embodiments of the second aspect, generating the second intermediate frame includes: generating a second intermediate frame corresponding to the second time parameter based on multiple historical rendering frames and the second time parameter.
[0020] In some embodiments of the second aspect, the step of generating a first intermediate frame corresponding to the first time parameter based on multiple historical rendering frames and the first time parameter is executed by the floating-point arithmetic unit of the graphics processing unit (GPU).
[0021] In some embodiments of the second aspect, the step of generating a second intermediate frame corresponding to the second time parameter based on multiple historical rendering frames and the second time parameter is performed by the floating-point arithmetic unit of the graphics processing unit (GPU).
[0022] In some embodiments of the second aspect, the step of determining the corresponding first target frame by repairing the pixel-missing regions of the first intermediate frame based on the first historical rendering frame is performed by the matrix operation unit of the graphics processing unit (GPU).
[0023] In some embodiments of the second aspect, wherein, while generating the second intermediate frame, determining the corresponding first target frame based on the first historical rendering frame by repairing the pixel missing areas of the first intermediate frame includes: the matrix operation unit of the graphics processing unit (GPU) receiving the first intermediate frame sent by the floating-point operation unit of the GPU; and while the floating-point operation unit of the GPU generates the second intermediate frame, the matrix operation unit of the GPU determines the corresponding first target frame based on the first historical rendering frame by repairing the pixel missing areas of the first intermediate frame.
[0024] According to a third aspect of this application, a chip is provided, comprising: a first arithmetic circuit for generating an intermediate frame corresponding to a time parameter based on a plurality of historical rendering frames and a time parameter; and a second arithmetic circuit for determining a corresponding target frame based on a first historical rendering frame by repairing pixel missing regions of the intermediate frame, wherein the target frame is located after the plurality of historical rendering frames.
[0025] In some embodiments of the third aspect, the intermediate frame includes a first intermediate frame and a second intermediate frame, and the target frame includes the first target frame. The method further includes: while the first arithmetic circuit generates the second intermediate frame, the second arithmetic circuit determines the corresponding first target frame by repairing the pixel-missing regions of the first intermediate frame. By processing the intermediate frame generation task and the repair task in parallel, computational resources can be fully utilized, task waiting time can be reduced, and thus the generation efficiency of multiple target frames can be significantly improved.
[0026] In some embodiments of the third aspect, the plurality of historical rendering frames includes a second historical rendering frame, the rendering time of the second historical rendering frame being later than the rendering time of other historical rendering frames in the plurality of historical rendering frames; the first historical rendering frame is one or more of the plurality of historical rendering frames, the first historical rendering frame being different from the second historical rendering frame; or the rendering time of the first historical rendering frame being earlier than the rendering time of each historical rendering frame in the plurality of historical rendering frames; or the first historical rendering frame is a plurality of historical rendering frames, some of the first historical rendering frames belonging to the plurality of historical rendering frames, and some of the first historical rendering frames being different from the plurality of historical rendering frames.
[0027] According to a fourth aspect of this application, a server is provided, comprising: a central processing unit (CPU), an accelerator, and a memory, wherein the accelerator executes instructions in the memory, causing the server to perform the method according to a first aspect of this application.
[0028] According to a fifth aspect of this application, a frame generation apparatus is provided, comprising: an intermediate frame generation unit and a target frame determination unit. The intermediate frame generation unit is configured to generate an intermediate frame corresponding to a time parameter based on multiple historical rendering frames and a time parameter. The target frame determination unit is configured to determine a corresponding target frame based on a first historical rendering frame by repairing pixel-missing regions in the intermediate frame, wherein the target frame is located after the multiple historical rendering frames.
[0029] According to a sixth aspect of this application, a computer-readable storage medium is provided that stores one or more computer instructions thereon, wherein one or more computer instructions are executed by a processor to cause the processor to perform the method according to a first aspect of this application.
[0030] According to a seventh aspect of this application, a computer program product is provided, including machine-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this application. Attached Figure Description
[0031] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0032] Figure 1 illustrates a schematic diagram of an example environment 100 in which the devices and / or methods according to embodiments of this application may be implemented;
[0033] Figure 2 shows a flowchart of a frame generation method according to an embodiment of this application;
[0034] Figure 3 illustrates a schematic diagram of motion estimation provided according to some embodiments of this application;
[0035] Figure 4 illustrates a schematic diagram of generating intermediate frames according to some embodiments of this application;
[0036] Figure 5 illustrates a schematic diagram showing that multiple pixels have the same mapping position according to some embodiments of this application;
[0037] Figure 6A illustrates a schematic diagram A showing the relationship between a first historical rendering frame, multiple historical rendering frames, and a second historical rendering frame provided according to some embodiments of this application;
[0038] Figure 6B illustrates a schematic diagram of the relationship between a first historical rendering frame, multiple historical rendering frames, and a second historical rendering frame provided according to some embodiments of this application.
[0039] Figure 6C illustrates a schematic diagram of the relationship between a first historical rendering frame, multiple historical rendering frames, and a second historical rendering frame provided according to some embodiments of this application.
[0040] Figure 6D illustrates a schematic diagram of the relationship between a first historical rendering frame, multiple historical rendering frames, and a second historical rendering frame provided according to some embodiments of this application.
[0041] Figure 7 illustrates a schematic diagram of pixel missing region repair provided according to some embodiments of this application;
[0042] Figure 8 shows a schematic diagram of an encoding-decoding structure provided according to some embodiments of this application;
[0043] Figure 9 illustrates a schematic diagram of repairing pixel missing regions according to some embodiments of this application;
[0044] Figure 10 illustrates a schematic diagram of the parallel execution target frame generation and repair process provided according to some embodiments of this application;
[0045] Figure 11 shows a schematic block diagram of a frame generation apparatus according to an embodiment of the present application;
[0046] Figure 12 shows a schematic diagram of the structure of a graphics processor provided according to some embodiments of this application; and
[0047] Figure 13 shows a structural block diagram of a server provided according to some embodiments of this application. Detailed Implementation
[0048] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments.
[0049] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " indicates "and / or," for example, A / B can mean A or B, or A and B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.
[0050] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.
[0051] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.
[0052] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0053] The numbers or values used in this specification are illustrative and are intended only to facilitate understanding of the technology of the embodiments of this application, and are by no means intended to limit the scope of this application.
[0054] As mentioned above, depending on the interpolation location, frame interpolation techniques can be divided into in-betweening and out-painting techniques. In-betweening involves inserting a filtered frame into a known sequence of consecutive frames. Out-painting involves predicting and generating a new frame outside the beginning or end of a known frame sequence. However, in-betweening requires waiting for the next frame to be rendered, introducing additional latency and making it unsuitable for devices with limited computing resources. Furthermore, existing out-painting techniques primarily rely on optical flow calculations and G-Buffer information in the deferred rendering pipeline, while typical rendering engines primarily use forward rendering, limiting the application of existing out-painting generation techniques on rendering devices.
[0055] Based on this, this application provides a frame generation method. This method can predict the motion trajectory of the next moment through the motion information of multiple historical rendered frames, thereby generating corresponding intermediate frames. The generation of these intermediate frames does not rely on optical flow information, thus reducing computational complexity and improving frame generation efficiency. Furthermore, after generating the intermediate frames, the first historical rendered frame can be used as a guide to finely repair pixel-missing areas or other changes in the intermediate frames, further enhancing the realism and smoothness of the image, thereby improving the visual effects of videos and animations.
[0056] The methods provided in this application can be applied to electronic devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.
[0057] Figure 1 illustrates a schematic diagram of an example environment 100 in which the devices and / or methods according to embodiments of this application may be implemented. As shown in Figure 1, the example environment 100 includes electronic device 102 and electronic device 104. Electronic device 102 can communicate with electronic device 104 via a network (wired or wireless, such as via wireless local area networks (WLAN) (e.g., wireless fidelity, Wi-Fi)) to send time parameters and related information to electronic device 104, which then generates and repairs the target frame. The electronic device can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device with image processing capabilities. It can also be a server (a single server or a server cluster consisting of multiple servers) or a cloud computing service center. This application does not limit the specific type of electronic device. For example, electronic device 104 can include or be implemented in any physical or virtual device with computing capabilities, such as servers, mainframes, general-purpose computers, virtual machines, terminal devices, cloud computing systems, etc. It should be noted that electronic device 104 can be a device with a Graphics Processing Unit (GPU), where a GPU is a type of dedicated hardware in a computer specifically designed for processing graphics. Its advantage lies in its ability to process multiple similar tasks in parallel; for example, when rendering an image, multiple pixels are rendered simultaneously, and a GPU can accelerate this process. A single GPU can process thousands of tasks in parallel.
[0058] In some embodiments, scene 106 may include one or more two-dimensional or three-dimensional models, each model consisting of multiple basic primitives (such as triangles, quadrilaterals, etc.). For example, models may include models or objects of shapes such as trees, light sources, vehicles, houses, buildings, geographical environments, people, or animals. In some embodiments, identifiers may be added to objects in scene 106. Each object may have a unique identifier. Scene rendering is the process of converting information such as mathematical models, materials, and lighting into two-dimensional pixel images. For example, by rendering scene 106, the corresponding historical rendering image 108, i.e., historical rendering frame, can be determined. During rendering, a virtual camera can be pre-determined, and then rendering can be performed from the perspective of a virtual machine. For example, when a user uses a game built with a three-dimensional scene, the user's display perspective can follow the movement of the character controlled by the user; the user's perspective is the perspective of the virtual camera, and the visible area displayed on the monitor is the current perspective area. In some embodiments, rendering may be performed on a server or on electronic devices such as electronic device 102 or electronic device 104. For example, it may be performed by a neural network model such as an AI model deployed on a server or electronic device. It is understood that a rendering frame and an image frame can be considered as different expressions of the same concept. They all represent a static image at a specific moment. In some embodiments, multiple rendered historical frames 108 can be stored in electronic device 102 or electronic device 104. The historical rendered frames 108 can have specific timestamps and resolution information for subsequent intermediate frame generation and repair.
[0059] As shown in Figure 1, the user can input time parameters through the electronic device 102 or select the desired time parameters from those provided by the electronic device 102. For example, the user can input the desired time parameters through an application programming interface (API) or a graphical user interface (GUI). It is understood that the user can input one or more time parameters simultaneously, with multiple times corresponding to the generation of multiple intermediate frames. The time parameter can be a specific point in time or a time range, and can be used to determine the position of the intermediate frame in the time series.
[0060] As shown in Figure 1, the electronic device 104 can generate the corresponding target frame based on time parameters and historical rendering frames 108.
[0061] For example, electronic device 104 can determine the motion trajectory of a target object, such as a vehicle or the pixels contained within a vehicle, based on multiple consecutive historical rendering frames 108, such as historical rendering frame A, historical rendering frame B, and historical rendering frame C (not shown in the figure). Historical rendering frame C can be the latest frame rendered. After determining the motion trajectory of the target object, the position of the target object at the time corresponding to the user-input time parameter can be predicted. Based on this, corresponding intermediate frames (not shown in the figure) can be generated by moving each pixel in historical rendering frame C to the predicted position.
[0062] As shown in Figure 1, when a car drives out from behind a tree, a portion of the tree may obscure the car, resulting in missing pixels for the car in the generated intermediate frame because there are no corresponding visible pixels in the current frame. Based on this, a first historical rendering frame (containing the object with the missing pixel region) can be determined from the historical rendering frames. Using this first historical rendering frame as a guide, the missing pixel regions or other changes in the intermediate frame can be finely repaired to obtain the target frame. For example, in historical rendering frame 108, a frame containing complete car information and not obscured by a tree can be selected as the first historical rendering frame. After determining the first historical rendering frame, the pixels corresponding to the car in the first historical rendering frame can be mapped to the missing pixel regions in the intermediate frame. Alternatively, the first historical rendering frame can be input into a pre-trained deep learning model, which can then repair the missing pixel regions in the intermediate frame.
[0063] It should be noted that the method provided in this application can be applied to some AR or VR applications, such as shopping, games, and projection. For example, the graphics rendering method provided in this application can be used by smart wearable devices (such as VR glasses / AR glasses), mobile terminals (such as mobile phones, tablets, etc.), or other devices that can support AR or VR applications. For instance, VR applications can be installed on wearable devices to provide services to users. VR applications can provide users with various 3D scenes, and users can view the 3D scenes in the VR application through the display screen of the wearable device, which is equivalent to giving users an immersive experience and improving the user experience.
[0064] The following is a flowchart of a frame generation method 200 according to an embodiment of this application, with reference to FIG2. It should be noted that the method 200 according to an embodiment of this application can be implemented, for example, at the electronic device 104 shown in FIG1. It should be understood that the method 200 may also include additional actions not shown, and the scope of this application is not limited in this respect.
[0065] As shown in Figure 2, at step 202, an intermediate frame corresponding to the time parameters is generated based on multiple historical rendering frames and time parameters. A rendering frame refers to the final result obtained by a graphics processing unit (GPU) through a series of complex calculations and algorithms, converting elements such as 3D models, textures, and lighting into a 2D image. For example, it can be an image rendered by the GPU, also known as an image frame. For instance, the rendering process of an image frame is essentially observing a 3D model from a certain position and displaying the observed scene as a 2D image, thus obtaining the image frame. Simply put, rendering a 3D model can be understood as taking a 2D image of the 3D model from a certain position using a camera. Therefore, after determining the observation point position in the 3D model, the objects to be rendered in the 3D model can be determined, thus determining the content in the 2D image. Playing multiple rendering frames continuously can create dynamic effects. For example, when multiple rendering frames are played continuously at a certain frame rate (such as 24 frames per second, 30 frames per second, 60 frames per second, etc.), the human eye will perceive a continuous dynamic effect.
[0066] In some embodiments, a historical rendering frame refers to a previously rendered image frame, i.e., an image frame that has already been rendered. Multiple historical rendering frames can be consecutive, for example, they can be the first, second, third, ... (n-1)th frame, the nth frame, etc., during game execution. Multiple historical rendering frames can also be non-consecutive, such as intermittently or randomly sampled historical rendering frames, for example, the first, third, fifth, ... (n-5)th frame, the (n-3)th frame, etc., during game execution.
[0067] The time parameter can be used to characterize the generation time of the frame to be generated, or to represent the relative position of the frame to be generated within a sequence of historical rendered frames (among multiple historical rendered frames). For example, the time parameter can be a value between 0 and 1, representing the position after multiple historical rendered frames. The time parameter can also be a timestamp, i.e., the position of the frame to be generated on the timeline. When the time parameter is a timestamp, the timestamp can be converted into a frame index based on the frame rate to calculate the relative position of the frame to be generated with multiple historical rendered frames. Here, the frame rate refers to the number of rendered frames generated per second. In some embodiments, the time parameter can be set by the user, for example, the user can input the corresponding time parameter, or the user can select a suitable time parameter from multiple default time parameters. This application does not specifically limit the setting of the time parameter.
[0068] After determining the time parameters and multiple historical rendering frames, corresponding intermediate frames can be generated through motion estimation. Motion estimation can involve determining the motion trajectory of pixels based on multiple historical rendering frames. Motion estimation can occur in pixel space or world space. For example, by fitting adjacent historical rendering frames, the displacement information of pixels between adjacent frames can be determined, thereby determining the continuous motion path of pixels on the time axis. Based on this, the position of a pixel at the time parameter can be predicted according to the motion trajectory and the time parameters. Based on the position of each pixel at the time parameter, corresponding intermediate frames can be generated. It can be understood that intermediate frames represent an intermediate state of the frame to be generated; subsequent processing of intermediate frames is needed to determine the final target generated frame.
[0069] As shown in Figure 2, at step 204, based on the first historical rendering frame, the corresponding target frame is determined by repairing the pixel-missing regions of the intermediate frames. This target frame is located after multiple historical rendering frames. The first historical rendering frame can be a rendered image frame, and there can be one or more first historical frames. The first historical rendering frame can be the same as or different from the aforementioned multiple historical rendering frames. "Same" can mean that multiple first historical rendering frames are partially the same as multiple historical rendering frames, or they can be completely the same. It is understood that, since only the motion information contained in multiple historical rendering frames is considered during the generation of intermediate frames without considering other factors, the relative movement of objects between adjacent frames may cause some areas to be occluded. For example, a foreground object moves to a certain area, causing that area to be visible in the current frame but occluded in the next frame, or a background area becomes visible for the first time due to foreground movement. In other words, there may be pixel-missing regions in the generated intermediate frames. Pixel-missing regions can refer to occluded areas (such as the area of a vehicle obscured by trees) that have no corresponding pixels in the intermediate frame, thus forming holes; pixel missing can also be understood as a pixel value of 0; for pixel-missing regions, at least some pixels in the pixel-missing region have a pixel value of 0. It is understandable that the missing pixel values in the pixel-missing areas of intermediate frames may exist in historical rendered frames. By utilizing information from historical rendered frames, the generation of repair content can be guided to some extent, improving the accuracy and naturalness of the repair. Based on this, the pixel-missing areas can be determined from the first historical rendered frame. For example, if the pixel-missing area in the intermediate frame is visible in the first historical rendered frame, its pixels can be directly copied. In other embodiments, a generative network can also be used to generate corresponding content based on the first historical rendered frame to repair the pixel-missing areas of the intermediate frame, thereby determining the corresponding target frame. By utilizing visible pixel information from historical rendered frames, the pixel-missing areas in the current frame can be filled more accurately, making the repaired image visually more coherent and realistic.
[0070] It is understood that the target frame is located after multiple historical rendering frames. In other words, the frame generation method provided in this application embodiment is an extrapolation frame method, which can predict the target frame (future frame) at the next moment based on multiple historical rendering frames, without waiting for the next frame to finish rendering. The generated target frame can be used to expand video or animation content and improve the rendering quality of video or animation.
[0071] In this way, the motion trajectory of the next moment can be predicted using motion information from multiple historical rendering frames, thereby generating corresponding intermediate frames. The generation of these intermediate frames does not rely on other complex information, thus reducing computational complexity and improving frame generation efficiency. Furthermore, after generating the intermediate frames, the first historical rendering frame can be used as a guide to finely repair missing pixel areas or other changes (such as shadow changes) in the intermediate frames, further enhancing the realism and smoothness of the image, thereby improving the visual effects of videos and animations.
[0072] In some embodiments, the foreground can be understood as the main subject included in the image, or the object that needs to be focused on. The foreground can also be understood as the region of interest (ROI) in the image. The background is the area in the image other than the foreground. For example, in an image that includes traffic lights, the foreground (or foreground object) is the area where the traffic lights are located, and the background is the area in the image other than the foreground.
[0073] In some embodiments, the second historical rendering frame can be the latest rendering frame among multiple historical rendering frames, with the latest rendering time, and can serve as a time reference. That is, the second historical rendering frame is the frame that is temporally closest to the current time (or the predicted time) among multiple historical rendering frames. Based on multiple historical rendering frames, motion trajectory fitting can be achieved. Motion trajectory fitting refers to establishing a mathematical model to describe the object's temporal motion patterns by analyzing the motion information in multiple historical rendering frames. Motion trajectory fitting can occur in pixel space (e.g., directly fitting the motion trajectory in the image coordinate system (pixel space)) or in world space (e.g., mapping pixel motion to a three-dimensional world coordinate system to fit the object's trajectory in real space). For example, the motion trajectory can be fitted by inverse projection of a depth map to world coordinates and then reprojected back to pixel space. After determining the object's motion trajectory based on multiple historical rendering frames, the object's motion vector at the next moment can be predicted based on time parameters, i.e., the motion vector of the intermediate frame relative to the second historical rendering frame. Here, the motion vector refers to the displacement vector of each pixel or feature point, describing the expected positional change of the pixel at the next moment. Based on this motion vector, the corresponding intermediate frame can be generated by mapping the pixels of the second historical rendering frame. For example, if the motion vector of a pixel is (dx, dy), then its position in the intermediate frame is (x+dx, y+dy). In other words, using the second historical rendering as the starting point for motion estimation can provide a high-quality reference image for the generation of intermediate frames, reduce error accumulation during the interpolation process, and ensure the accuracy of the motion vector.
[0074] In some embodiments, the generation time of the target frame indicated by the time parameter can be located between the rendering time corresponding to the second historical rendering frame and the rendering time corresponding to the third historical rendering frame. The third rendering frame can be the rendering frame to be generated at the next moment, i.e., a future rendering frame. It can be understood that the third rendering frame is generated according to a preset frame rate, meaning the time interval between the third rendering frame and the second historical rendering frame is fixed. It should be noted that the time parameter can be one or multiple. When there is only one time parameter, only one target frame is generated, located between the third rendering frame and the second historical rendering frame. When there are multiple time parameters, multiple target frames are generated. Multiple target frames can be sequentially inserted between the third rendering frame and the second historical rendering frame based on the order of the time parameters. For example, the value range of the time parameter t can be [0,1], used to represent the relative position of the target frame between the second historical rendering frame (t=0) and the third rendering frame (future frame) (t=1). When t is 0.5, a target frame between two frames can be generated; when t = [0.25, 0.5, 0.75], three target frames between two frames can be generated, thereby increasing the original frame rate by 3 times. It should be noted that multiple time parameters can be uniform or non-uniform. For example, the time parameter t can also be t = [0.1, 0.3, 0.9].
[0075] By setting different time parameters, multiple target frames can be generated between two frames, thereby increasing the frame rate and concentrating computing power on key frame generation, achieving the effect of exchanging low computing power for high frame rate.
[0076] In some embodiments, determining the motion vector is essentially fitting the motion trajectory. When fitting the motion trajectory, a suitable fitting model can be selected. The fitting model can be a function or model found through mathematical methods that characterizes the potential patterns or trends in the data and predicts the results of new data based on historical data, minimizing the error with the given dataset. This model can be a simple linear relationship or a complex nonlinear relationship. For example, the fitting model can be a single function type, such as a linear function (suitable for uniform motion), a quadratic function (suitable for uniformly accelerated motion), or a higher-order function (suitable for complex nonlinear motion). The fitting model can also be a combination of functions, such as a combination of linear functions with linear functions, a combination of quadratic functions with quadratic functions, a combination of linear functions with quadratic and higher-order functions, or a combination of quadratic and higher-order functions. Alternatively, the combined function can be a piecewise function, using different functions for different time periods. By selecting a suitable fitting model, the motion vector of the intermediate frame relative to the second historical rendering frame can be determined. For example, if pixel A is at position (x1, y1) in frame t-1 and at position (x2, y2) in frame t, the position in frame t+1 (the intermediate frame) can be predicted by fitting the linear model x(t) = a × t + b. It should be noted that the motion vector can be a motion vector matrix or a motion vector array. Each pixel in the image corresponds to a motion vector, which is the direction and distance of each pixel's movement from the current frame (the second historical rendering frame) to the next frame (the intermediate frame).
[0077] Figure 3 illustrates a schematic diagram of motion estimation provided according to some embodiments of this application. As shown in Figure 3, the ball (pixel or object) at time t i-2 At time t i-1 The displacement within is m i-1 At time t i-1 At time t i The displacement within is m i At time t i At time t i+α The displacement within is m i+α α can be a time parameter input by the user. At time t i-1 The corresponding rendering frame is historical rendering frame 302, at time t i The corresponding rendering frame is historical rendering frame 304, at time t. i+α The corresponding predicted frame is intermediate frame 306. Assuming the pixel's motion between adjacent historical rendering frames is uniformly accelerated, the fitting model can be a uniformly accelerated motion model. If the time interval between adjacent historical rendering frames, such as historical rendering frame 302 and historical rendering frame 304, is t, then the acceleration of the sphere (pixel or object)... It is the following formula (1):
[0078] It can be understood that the initial velocity of historical rendering frame 302 is: The initial velocity of historical rendering frame 304 is: Based on this, and using a uniformly accelerated motion model, the position p of the ball in the intermediate frame is... i+α It can be expressed as equation (2):
[0079] Where, p i For the ball at time t i Position, p i-1 For the ball at time t i-1 The location. and m i-1 Substituting into equation (2), we can obtain the time t of the ball. i+α Position p i+α It is the following formula (3):
[0080] In some embodiments, inverse projection using the camera pose (camera intrinsic matrix) and depth map can determine the world space coordinates corresponding to the ball's position at different times. Based on the world space coordinates corresponding to multiple positions, the ball's trajectory can be fitted as a straight line, thereby predicting the world space coordinates of the ball in intermediate frame 306. The camera intrinsic matrix can then be projected onto pixel space to obtain the ball's position at time t. i+α Position p i+α It is the following formula (4):
[0081] p i+α =α(p i -p i-1 )+p i (4)
[0082] In some embodiments, after determining the motion vector of the intermediate frame relative to the second historical rendering frame, the corresponding intermediate frame can be generated based on the second historical rendering frame and the motion vector. For example, the mapping position corresponding to each pixel in the second historical rendering frame can be determined according to the motion vector, and the corresponding intermediate frame can be generated based on the mapping result. In some embodiments, the intermediate frame can be generated by mapping the pixels of the current frame to the corresponding positions according to the motion vector through a warp operation (e.g., image warping or pixel remapping). For example, the mapping positions of all pixels can be combined to generate an intermediate frame. The warp operation can be a geometric transformation of the image using a transformation matrix, such as translation (moving the image a certain distance in the horizontal or vertical direction), rotation around a point (rotating the image around a center point by a certain angle), scaling (changing the size of the image, making it larger or smaller), etc., to change the shape and position of the image. In some embodiments, the warp operation is usually implemented by bilinear interpolation or bicubic interpolation to handle non-integer pixel displacements.
[0083] Figure 4 illustrates a schematic diagram of generating intermediate frames according to some embodiments of this application. As shown in Figure 4, the movement position and direction of each pixel in the second rendering frame 402 at the next moment can be determined by the motion vector. Based on the movement position and direction, each pixel can be mapped to its corresponding position. Based on the mapped positions of each pixel, a new image such as intermediate frame 404 can be formed. In this way, by using the second historical rendering frame as a basis and generating intermediate frames by transforming the second historical rendering frame, existing rendering resources can be reused, redundant calculations can be avoided, and computational resources and time can be saved. For complex nonlinear motion (such as curvilinear motion, accelerated motion, etc.), the transformation operation can also more flexibly adjust the pixel position to match the actual motion trajectory, thereby generating more accurate intermediate frames. In addition, by processing the motion vector of each object separately, it can be ensured that the relative positions and motion relationships between pixels are correctly maintained, so that the generated image is more consistent with the laws of the real world.
[0084] In practical applications, when moving pixels (such as during the generation of intermediate frames via warp operations), pixels from different objects may move to the same position due to motion estimation, occlusion relationships, or dynamic scene changes. In this case, it's necessary to determine which pixel should remain at that position to avoid pixel conflicts that reduce the accuracy of intermediate frame generation. Based on this, the depth of the object can be used to determine which pixel needs to be moved to that position. For example, the depth map can be used to determine whether an object belongs to the foreground or background. The depth map records the distance of each pixel to the camera; the smaller the depth, the closer the object is to the camera. That is, when multiple pixels are mapped to the same position, the pixel with the smallest depth value (i.e., the object closest to the camera) can be retained, ensuring that the generated intermediate frame follows various rules of the physical world, such as occlusion rules and motion rules. For example, if the pixels of a vehicle (foreground, depth 0.5) and trees (background, depth 0.8) overlap in an intermediate frame, the vehicle's pixels can be retained based on the depth comparison. This method enhances the visual realism and generation accuracy of the intermediate frame.
[0085] In other embodiments, the type and dynamic attributes of the object can be combined to determine which pixels need to be retained or displayed. Based on depth comparison, different priorities are assigned to different types of objects. For example, the pixels to be retained can be determined based on the pixel's depth, a first weight value corresponding to that depth, the priority of the object type corresponding to the pixel, and a second weight value corresponding to that priority. Alternatively, when multiple pixels correspond to the same mapping position and the same depth, the pixels to be retained can be determined based on the priority of the pixel's object type. Table 1 below shows the priorities corresponding to each object type. As shown in Table 1, the dynamic foreground has the highest priority, and the static background has the lowest priority.
[0086] Table 1 Priority Table for Object Types
[0087] It's understandable that in practical applications, fast-moving objects like bullets might result in inaccurate depth values recorded in the depth map due to errors in their motion vectors. Therefore, the depth tolerance can be dynamically adjusted based on the object's speed. The depth tolerance refers to the allowable error range during depth testing, used to handle minor differences in depth values caused by motion or other factors. In one example, if two or more pixels compete for the same display position, and their depth value difference is within the depth tolerance range, they can be considered "approximately the same depth," requiring consideration of other factors to determine which pixels should be retained.
[0088] Figure 5 illustrates a schematic diagram of multiple pixels having the same mapping position according to some embodiments of this application. As shown in Figure 5, during the warp operation, two or more pixels in the second historical rendering frame 502 may be mapped to the same position in the intermediate frame 506. For example, pixel A and pixel B have the same mapping position. To avoid pixel conflicts or pixel overlay problems, the pixels to be retained, i.e., the pixels to be displayed in the intermediate frame, can be determined based on the depth values corresponding to the two pixels. For example, the pixels to be retained can be determined based on the depth values corresponding to pixel A and pixel B in the depth map 504. If the depth value corresponding to pixel A is less than the depth value corresponding to pixel B, pixel A can be retained, i.e., the pixel can be mapped to the corresponding position in the intermediate frame. In this way, when pixel conflicts occur, the pixel closest to the camera can be selected for display, thereby generating a more realistic and reasonable image.
[0089] In practical applications, scenes can contain both dynamic and static objects. Due to the relative positions of static and dynamic objects, occlusion can lead to missing pixels. For example, between consecutive frames, some areas may be occluded or exposed by other objects. For instance, when a car drives out from behind a tree, part of the tree may obscure the car, resulting in missing pixels for that part of the car in the generated intermediate frame because there are no corresponding visible pixels in the current frame. Alternatively, during the generation of intermediate frames, the estimated motion vectors may not be entirely accurate, especially in scenes with complex motion or rapid changes. This can cause pixels to be incorrectly mapped to inappropriate locations, resulting in holes. It's understandable that missing pixels not present in the current rendering frame may exist in previous rendering frames; that is, previous rendering frames can guide the generation of repaired content to some extent. Based on this, the first historical rendering frame can be used to repair areas with missing pixels, thereby determining a more realistic target frame. The first historical rendering frame differs from the second historical rendering frame; it can be a frame preceding the second historical rendering frame. There can be one or more first historical rendering frames.
[0090] The following examples, illustrated with Figures 6A-6D, illustrate the relationship between the first historical rendering frame, multiple historical rendering frames, and the second historical rendering frame. The multiple historical rendering frames serve as the basis for determining the motion vectors of the frame to be generated, while the second historical rendering frame is the most recent rendering frame. In the examples of Figures 6A-6D, the striped boxes represent the first historical rendering frame, and the rectangular boxes represent the historical rendering frames and the second historical rendering frame. In Figure 6A, the number of first historical rendering frames can be one, and this first historical rendering frame can be earlier than the multiple historical rendering frames and also earlier than the second historical rendering frame. That is, the first historical rendering frame used to repair missing pixel regions is earlier than the multiple historical rendering frames used for motion estimation. In Figure 6B, the number of first historical rendering frames is also one, and this first historical rendering frame can be one of the multiple historical rendering frames used to generate intermediate frames. For example, it can be the (n+2)th historical rendering frame or the (n+3)th historical rendering frame; this application does not impose any limitations on this. That is, the first historical rendering frame used to repair missing pixel regions can be one of the multiple historical rendering frames used to generate intermediate frames.
[0091] In Figure 6C, the number of first historical rendering frames can be multiple, for example, k. Some of these first historical rendering frames can be the same as multiple historical rendering frames, while others can be different. For example, the first historical rendering frames m+k-3, m+k-2, m+k-1, and m+k can be the same as the historical rendering frames n, n+1, n+2, and n+3, respectively. It should be noted that the first historical rendering frames cannot be the same as the second historical rendering frames. In Figure 6D, the number of first historical rendering frames can also be multiple, and all of these first historical rendering frames must be different from all other historical rendering frames; that is, all of these first historical rendering frames must be earlier than all other historical rendering frames.
[0092] In some embodiments, historical rendering frames record the state of the scene at different times, which may include visible information about missing pixels in intermediate frames. If the missing pixel region in the intermediate frame is visible in the first historical rendering frame, its pixels can be directly copied. Alternatively, the missing pixel region in the frame can be repaired using a pre-trained inpainting model (e.g., a pre-trained deep learning model such as a Generative Adversarial Network (GAN) or Variational Autoencoder (VAE)) to determine the corresponding target frame. Since there is a certain temporal continuity between historical rendering frames and intermediate frames, they are similar in scene structure and object features. Using the first historical rendering frame to repair the missing pixel region in the intermediate frame can leverage the pixel features and contextual information in the historical frame to guide the generation of the repaired content, making the repaired target frame more realistic. In other embodiments, after generating the intermediate frame, the missing pixel content can be inferred from the surrounding pixels of the intermediate frame. For example, the missing content can be inferred by analyzing the pixel features of the non-missing regions in the intermediate frame and using interpolation, diffusion, or convolution operations, thereby repairing the missing pixel region in the intermediate frame. Alternatively, the position and shape of the occluded area can be predicted by extrapolating the physical laws of the motion trajectory, thereby repairing the pixel-missing areas of the intermediate frame and generating the corresponding intermediate frame.
[0093] In some embodiments, a VAE (Visual Encoding Environment) is a generative model primarily used to learn latent representations of data and generate new data similar to the original data distribution. Its core idea is to map input data to a latent space through an encoder, and then reconstruct or generate data from the latent space through a decoder. GANs (Generative Analog Arrays) can be based on game theory, generating data through adversarial training between a generator and a discriminator. The generator attempts to produce realistic data, while the discriminator attempts to distinguish between generated and real data.
[0094] The following uses a GAN as an example to illustrate the process of repairing missing pixel regions. Figure 7 shows a schematic diagram of pixel missing region repair according to some embodiments of this application. As shown in Figure 7, the generated intermediate frame 702 can be input into the GAN model as the input of the first part of the GAN model. Similarly, the first historical rendering frame 708 can also be input into the GAN network as the input of the second part of the GAN model. The number of first historical rendering frames 708 can be one or more. The GAN model may include a generator and a discriminator 714. The generator may include an encoder 704, an encoder 710, and a decoder 706. The encoder receives the input sequence and converts it into a fixed-length context vector. This vector is an internal representation of the input sequence, capturing the key features of the input information. For example, encoder 704 can be used to extract image features of intermediate frame 702, and encoder 710 can be used to extract image features of first historical rendering frame 708. For example, encoder 710 can be used to capture the temporal dynamic features (such as motion trajectory and lighting changes) of the first historical rendering frame 708 to compensate for the missing context information in the intermediate frame 702. The context vectors of the two encoders can be fused through channel concatenation, element-wise addition, or attention mechanisms to ensure that spatial and temporal features are complementary. Decoder 706 can generate the corresponding target frame 712 based on the fused feature vector. For example, decoder 706 can upsample and reconstruct the fused feature vector to generate the repaired target frame 712. In some embodiments, upsampling can be performed using methods such as deconvolution (transposed convolution) or bilinear interpolation to restore the resolution and detail information of the image.
[0095] It is understandable that there may be missing pixel regions in the intermediate frame 702, which can be repaired using the first historical rendering frame 708. In other words, the GAN model can extract features from the intermediate frame 702 and the first historical rendering frame 708 (the current spatial features of the intermediate frame 702 + the temporal dynamic features of the first historical rendering frame 708) using a dual encoder, and combine this with a dynamic feature selection mechanism to generate a high-quality and spatiotemporally coherent target frame 712. After generating the target frame 712, the discriminator 714 is responsible for distinguishing the generated target frame 712 from the real Ground Truth image 716.
[0096] It's important to note that a typical encoder, acting as a convolutional network, processes all pixels equally, including invalid (missing) pixels. However, the missing pixel regions in intermediate frame 702 are generally irregular. Therefore, to automatically focus on valid pixel regions and suppress noise propagation in missing areas, the encoder can be a gated convolutional network. A gated convolutional layer can include two parallel convolutional operations: one branch generates a feature map, and the other generates a gating map. The gating map indicates the importance of each location in the feature map. Gated convolution provides a learnable feature selection mechanism by generating a mask through a learnable gating branch. The mask value is between 0 and 1, representing the importance of each pixel location. By dynamically weighting the input feature map with a mask, gated convolution can dynamically select and retain valid feature information while suppressing or ignoring invalid (missing) pixel information. This helps the GAN model better focus on valid image regions, improving the accuracy of the inpainting. In other words, compared to standard convolution, gated convolution can handle irregular missing regions more flexibly, improving the robustness of the inpainting.
[0097] As shown in Figure 7, skip connections can be introduced into the decoder, which allows for direct reuse of multi-scale features from the encoder, avoiding loss of detail. Additionally, low-level features can be preserved, improving detail recovery capabilities. In some embodiments, a mask image of the missing region can be introduced as additional input to guide the GAN model to focus on the effective region, avoiding interference from missing pixels.
[0098] In some embodiments, the training and inference processes of a GAN model are similar. The loss function can include two aspects: a discriminator loss function and an absolute error loss function (L1 loss function). The discriminator loss function is used to train the discriminator to more accurately distinguish between generated and real images. The absolute error loss function measures the pixel-level difference between the generated image and the ground truth image. By minimizing the L1 loss function, the generated image can be made closer to the real image at the pixel level.
[0099] Figure 8 shows a schematic diagram of an encoder-decoder structure provided according to some embodiments of this application. As shown in Figure 8, the encoder 802 and decoder 804 can constitute the Encoder-Decoder structure of the above-described GAN model. The encoder 802 uses a convolutional kernel W... m By performing a convolution operation on the input data X, we can obtain the feature map M, as shown in equation (5):
[0100] M = Conv(W m,X) (5)
[0101] Decoder 802 uses another convolutional kernel W f The input data X is convolved to obtain the gated graph F, as shown in equation (6):
[0102] F = Conv(W) f ,X) (6)
[0103] Based on this, the feature map M can be processed by the activation function σ, and then multiplied element-wise with the gated map F to obtain the final output O. The activation function σ is typically used to introduce nonlinear factors to enhance the model's expressive power. Element-wise multiplication is a feature fusion method that combines the feature information from M and F to generate the final output. For example, the output O can be determined using the following equation (7):
[0104] O=σ(M)⊙F (7)
[0105] It is understandable that the shape or size of shadow areas in an image may change with the position of the light source and the occlusion relationship of objects. A shadow area can refer to a dark area formed due to the occlusion of a light source by an object, exhibiting a difference in brightness / color compared to the surrounding area. For example, when a light source (such as natural light or artificial light) is partially or completely blocked by an object, the light cannot directly illuminate the area behind the object, resulting in a significant reduction in the intensity of reflected light in that area, forming a shadow area. Based on this, to ensure higher image quality and greater realism in the generated target frame, the shadow area of the target frame can be adjusted according to the shadow changes in consecutive historical rendering frames. For example, the morphological parameters of the shadow area in the target frame can be predicted based on the morphological change curves of historical rendering frames. In some embodiments, the morphological parameters of the shadow area in historical rendering frames, such as contour, area, and aspect ratio, can be determined by segmenting the shadow area. In some examples, the shadow area can be segmented based on color thresholds and depth information. Based on this, a curve showing the change of morphological parameters over time can be fitted using the segmentation results. For example, the curve showing the change of morphological parameters over time can be determined using polynomial fitting, Fourier transform, or other methods. By combining the time parameter and the curve, the shape parameters corresponding to the shadow area in the target frame can be predicted. In some embodiments, a deformation model (such as mesh-based deformation) can also be used to smoothly transition the shadow shape of historical rendering frames to intermediate frames.
[0106] By analyzing the shadow shape changes in historical frames, this method allows for the dynamic adjustment of shadow areas in intermediate frames, enabling shadows to transition naturally with the movement of objects or changes in lighting. For example, when an object moves, the shadow position, shape, and edge blur can be matched in real time, thus avoiding the abruptness of static shadows. Furthermore, it allows for shadow adjustments in the target frame to better conform to actual physical laws, enhancing the realism of the scene.
[0107] Figure 9 illustrates a schematic diagram of repairing missing pixel regions according to some embodiments of this application. As shown in Figure 9, a corresponding intermediate frame 904 can be generated based on the second historical rendering frame 902 and the motion vector. Due to the movement of the vehicle, part of the vehicle is obscured by trees in the previous moment and exposed in the next moment, resulting in a missing area of the vehicle in the generated intermediate frame 904. Based on this, the intermediate frame 904 can be repaired using the first historical rendering frame (not shown in the figure) to obtain the final target frame 906. The repair process can refer to the repair process described in the above embodiments, and will not be repeated here.
[0108] In some embodiments, the generation and repair of intermediate frames can be performed by the GPU. The GPU may include multiple processing units, which are assigned specific tasks based on application requirements to improve overall computational performance. For example, it may include floating-point units (FPUs) and matrix operation units (Tensor Cores). The FPU is the core component of the GPU responsible for handling floating-point calculations (such as 32-bit single-precision and 64-bit double-precision), and can be used for arithmetic operations (such as addition, subtraction, multiplication, division, square root, etc.), function operations (such as trigonometric functions (sin, cos), exponential functions (exp), logarithmic functions (log), etc.), and precision control (such as switching between single-precision (FP32) and double-precision (FP64) modes according to application requirements). The matrix operation unit is a hardware acceleration module designed specifically for deep learning, primarily performing matrix multiplication and accumulation (MMA) operations. Based on the characteristics of the multiple processing units, different types of frame generation tasks can be assigned to appropriate processing units, fully utilizing the GPU's parallel processing capabilities. In some embodiments, intermediate frame generation requires high-precision floating-point operations to ensure the accuracy of motion estimation and interpolation. Using an FPU can meet this requirement and avoid image artifacts caused by insufficient precision. For example, motion estimation (such as trajectory fitting) during intermediate frame generation relies on high-precision floating-point operations. Meanwhile, matrix operation units can process large amounts of data simultaneously, making them suitable for tasks like repairing missing pixels that require processing the entire image region, thus accelerating processing speed. For example, when using a convolutional neural network (CNN) to repair missing pixel regions, inter-layer weight matrix multiplication can be accelerated by Tensor Cores. That is, the floating-point operation unit can generate corresponding intermediate frames based on time parameters and multiple historical rendering frames; the matrix operation unit then repairs the intermediate frames generated by the floating-point operation unit to determine the corresponding target frame. In this way, intermediate frame generation is completed by the FPU, avoiding uneven load distribution across general-purpose computing units. Matrix-intensive tasks (such as AI model inference) are performed by Tensor Cores, reducing power consumption. This allows for maintaining high image quality of generated frames while reducing power consumption.
[0109] It's worth noting that the shared memory and registers within the GPU allow the floating-point and matrix operation units to directly access intermediate results, avoiding data transfer bottlenecks between the FPU and the matrix operation units. For example, the floating-point calculation results generated in intermediate frames are directly cached in the GPU's video memory for use by the Tensor Cores.
[0110] In some embodiments, to further improve the generation efficiency of the target frame, the generation and repair of intermediate frames can be performed in parallel. For example, the repair of the previous intermediate frame can be completed simultaneously with the generation of the current intermediate frame (here, the current intermediate frame and the previous intermediate frame do not refer to their temporal relationship, but rather to the different order in which they were generated). This allows for parallel processing of different tasks. Simultaneously, resource allocation is more rational, avoiding overload of a single unit. For example, the user can simultaneously input multiple time parameters, such as time parameter 1, time parameter 2, time parameter 3, etc. Intermediate frame 1 is generated based on time parameter 1 and multiple historical rendering frames. While repairing the pixel missing areas of intermediate frame 1 using a second historical rendering frame, the corresponding intermediate frame 2 can be generated based on time parameter 2 and multiple historical rendering frames (which may be the same as or different from the aforementioned historical rendering frames). Similarly, while repairing the pixel missing areas of intermediate frame 2 using a second historical rendering frame (which may be the same as or different from the aforementioned second historical rendering frame), intermediate frame 3 can be generated based on time parameter 3 and multiple historical rendering frames. And so on, multiple frame generation tasks that originally needed to be executed sequentially can be performed simultaneously, greatly shortening the overall generation process time. By processing intermediate frame generation and repair tasks in parallel in this way, computing resources can be fully utilized, task waiting time can be reduced, and the generation efficiency of the target frame can be significantly improved.
[0111] In some embodiments, computing resources can be dynamically allocated based on task complexity, priority, resource usage, and computational requirements to avoid overloading a single computing unit. For example, more computing resources can be allocated to computationally intensive intermediate frame generation tasks, while fewer resources can be allocated to relatively simple repair tasks. This improves resource utilization and reduces the overall energy consumption of the system.
[0112] In some embodiments, intermediate frames can be generated by the floating-point arithmetic unit and repaired by the matrix arithmetic unit simultaneously. These two processes can be pipelined, improving overall throughput. It can be understood that the generation and repair of intermediate frames are parallel processing methods with a time difference; that is, while generating the second intermediate frame, the first intermediate frame is repaired to obtain the first target frame. This time-difference parallel processing means that the generation and repair tasks overlap in time, not necessarily being completely synchronized. For example, the start time of generating the second intermediate frame may partially overlap with the end time of repairing the first intermediate frame, forming a pipeline stage. In other words, while the FPU is generating the second intermediate frame, the Tensor Core is simultaneously processing the repair of the first intermediate frame to generate the first target frame. This overlap of the two tasks in time forms a pipeline, reducing overall processing time. This approach improves resource utilization because while the FPU is processing the current frame, the Tensor Core is not idle but is processing the repair task of the previous frame, thus avoiding hardware unit idleness and improving overall throughput.
[0113] Figure 10 illustrates a schematic diagram of the parallel execution of target frame generation and repair processes according to some embodiments of this application. As shown in Figure 10, the GPU 1000 may include a floating-point arithmetic unit 1002 and a matrix arithmetic unit 1004. The floating-point arithmetic unit 1002 can generate a corresponding intermediate frame i-1 based on a time parameter i-1 and multiple historical rendering frames. Similarly, the floating-point arithmetic unit 1002 can also generate a corresponding intermediate frame i based on a time parameter i and multiple historical rendering frames. It should be noted that the multiple historical rendering frames used to generate intermediate frame i and intermediate frame i-1 can be all the same, or partially the same and partially different. While the floating-point arithmetic unit 1002 generates intermediate frame i, the matrix arithmetic unit 1004 can repair intermediate frame i-1 using a first historical rendering frame to determine the corresponding target frame i-1. Similarly, while the floating-point arithmetic unit 1002 generates intermediate frame i+1, the matrix arithmetic unit 1004 can repair intermediate frame i using a first historical rendering frame to determine the corresponding target frame i. It's understandable that the first historical rendering frame used to repair different intermediate frames can be the same or different. It's worth noting that the floating-point unit 1002 is suitable for handling large amounts of parallel computation; for example, generating intermediate frames may require a significant amount of pixel-level computation, such as motion estimation and interpolation for each pixel. The matrix operation unit 1004, on the other hand, is better suited for operations like convolution and matrix multiplication; for instance, repairing missing pixels may require deep learning models, which typically rely on matrix operations. Separating the repair and generation of intermediate frames leverages the strengths of both the floating-point unit 1002 and the matrix operation unit 1004, improving frame generation efficiency.
[0114] Figure 11 shows a schematic block diagram of a frame generation apparatus 1100 according to an embodiment of the present application. The frame generation apparatus 1100 can be implemented in an electronic device 104. The frame generation apparatus 1100 includes an intermediate frame generation unit 1110 and a target frame determination unit 1120. The intermediate frame generation unit 1110 is configured to generate an intermediate frame corresponding to a time parameter based on a plurality of historical rendering frames and a time parameter. The target frame determination unit 1120 is configured to determine a corresponding target frame based on a first historical rendering frame by repairing pixel missing regions of the intermediate frame, wherein the target frame is located after the plurality of historical rendering frames.
[0115] In some embodiments, the intermediate frame includes a first intermediate frame and a second intermediate frame, and the frame generation device 1100 is further configured to: determine the corresponding first target frame by repairing the pixel missing area of the first intermediate frame while generating the second intermediate frame.
[0116] In some embodiments, the intermediate frame generation unit 1110 may generate an intermediate frame corresponding to a time parameter based on multiple historical rendering frames and a time parameter via the floating-point arithmetic unit of the graphics processor.
[0117] In some embodiments, the target frame determination unit 1120 may receive an intermediate frame sent by a floating-point arithmetic unit via a matrix arithmetic unit of a graphics processor; and determine the corresponding target frame by repairing the pixel missing areas of the intermediate frame based on a first historical rendering frame via a matrix arithmetic unit.
[0118] In some embodiments, the plurality of historical rendering frames includes a second historical rendering frame, the rendering time of the second historical rendering frame being later than the rendering time of other historical rendering frames in the plurality of historical rendering frames.
[0119] In some embodiments, the first historical rendering frame is one or more of a plurality of historical rendering frames, and the first historical rendering frame is different from the second historical rendering frame; or the rendering time of the first historical rendering frame is earlier than the rendering time of each of the plurality of historical rendering frames; or there are a plurality of first historical rendering frames, some of which belong to the plurality of historical rendering frames, and some of which are different from the plurality of historical rendering frames.
[0120] In some embodiments, the time parameter is used to indicate one or more times between the rendering time of the second historical rendering frame and the rendering time of the third rendering frame, the third rendering frame being the next rendering frame to be generated according to a preset frame rate.
[0121] In some embodiments, the intermediate frame generation unit 1110 is further configured to: determine the motion trajectory of the target object based on multiple historical rendering frames, wherein the target object is located in the target scene and the multiple historical rendering frames are obtained by rendering the target scene; and generate a corresponding intermediate frame based on the motion trajectory and the second historical rendering frame.
[0122] In some embodiments, the intermediate frame generation unit 1110 is further configured to: determine a corresponding motion vector based on the motion trajectory and time parameters, wherein the time parameters are used to characterize the generation time of the intermediate frame; determine the mapping positions corresponding to multiple pixels in the second historical rendering frame based on the motion vector; and generate a corresponding intermediate frame based on the mapping positions corresponding to the multiple pixels.
[0123] In some embodiments, the frame generation apparatus 1100 further includes a shadow area adjustment unit, configured to: determine shape change information of the corresponding shadow area based on multiple consecutive historical rendering frames; and adjust the shadow area in the target frame based on the shape change information.
[0124] In some embodiments, the target scene is rendered by a graphics processor to obtain multiple historical rendering frames and / or a first historical rendering frame.
[0125] It should be noted that the frame generation device 1100 can be deployed in a graphics processor to generate a target frame following multiple historical rendering frames. The specific generation process can be found in the frame generation methods described in the above embodiments, and will not be elaborated upon here.
[0126] Some embodiments of this application provide a chip including a first arithmetic circuit and a second arithmetic circuit. The first arithmetic circuit is used to generate an intermediate frame corresponding to a time parameter based on multiple historical rendering frames and a time parameter. The second arithmetic circuit is used to determine a corresponding target frame based on the first historical rendering frame by repairing pixel-missing regions in the intermediate frame. The target frame is located after the multiple historical rendering frames. The chip can be a graphics processing unit (GPU), a neural network processing unit (NPU), or other types of processors or digital processing chips. This application does not limit the type of chip. When the chip is a GPU, the first arithmetic circuit can be a floating-point arithmetic unit, and the second arithmetic circuit can be a matrix arithmetic unit.
[0127] Figure 12 illustrates a schematic diagram of a graphics processor according to some embodiments of this application. As shown in Figure 12, the GPU 1200 may include a floating-point arithmetic unit 1210 and a matrix arithmetic unit 1220. The floating-point arithmetic unit can be used to generate intermediate frames corresponding to time parameters based on multiple historical rendering frames and time parameters. The matrix arithmetic unit can be used to determine the corresponding target frame based on a first historical rendering frame by repairing pixel missing regions in intermediate frames, wherein the target frame is located after multiple historical rendering frames.
[0128] The GPU 1200 according to the embodiments of this application can be used to execute the methods described in the embodiments of this application. The above and other operations and / or functions of each module in the GPU 1200 are respectively for implementing the corresponding processes of the methods provided in the above embodiments. For the sake of brevity, they will not be described again here.
[0129] It should also be noted that the embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0130] Some embodiments of this application provide a server including a central processing unit (CPU), an accelerator, and memory. The CPU can be used to acquire multiple historical rendering frames, or it can be used to acquire a first historical rendering frame. The accelerator can execute instructions in the memory, enabling the server to perform the frame generation methods described in the various embodiments above.
[0131] Figure 13 shows a structural block diagram of a server provided according to some embodiments of this application. As shown in Figure 13, the server 1300 may include a CPU 1310, an accelerator 1320, and a memory 1330. The memory 1330 may include volatile memory, such as random access memory (RAM). The memory 1330 may also include non-volatile memory, such as read-only memory (ROM), flash memory, HDD, or SSD. The CPU 1310, accelerator 1320, and memory 1330 can communicate with each other via a bus 1340. The bus 1340 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 1340 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 13, but this does not mean that there is only one bus or one type of bus.
[0132] This application may be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this application.
[0133] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0134] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0135] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.
[0136] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0137] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0138] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0140] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A method for frame generation, comprising: Based on multiple historical rendering frames and time parameters, an intermediate frame corresponding to the time parameters is generated. as well as Based on the first historical rendering frame, the corresponding target frame is determined by repairing the pixel missing areas of the intermediate frames, and the target frame is located after the plurality of historical rendering frames.
2. The method according to claim 1, wherein the intermediate frame includes a first intermediate frame and a second intermediate frame, and the method further includes: While generating the second intermediate frame, the corresponding first target frame is determined by repairing the pixel missing areas of the first intermediate frame.
3. The method according to claim 1, wherein generating an intermediate frame corresponding to the time parameter based on multiple historical rendering frames and a time parameter comprises: The intermediate frame corresponding to the time parameter is generated by the floating-point arithmetic unit of the graphics processor based on multiple historical rendering frames and time parameters.
4. The method according to claim 3, wherein determining the corresponding target frame by repairing the pixel-missing regions of the intermediate frames based on the first historical rendering frame includes: The intermediate frame sent by the floating-point arithmetic unit is received via the matrix arithmetic unit of the graphics processor; as well as The corresponding target frame is determined by repairing the pixel-missing areas of the intermediate frame based on the first historical rendering frame through the matrix operation unit.
5. The method according to claim 1, wherein the plurality of historical rendering frames includes a second historical rendering frame, the rendering time of the second historical rendering frame being later than the rendering time of other historical rendering frames in the plurality of historical rendering frames.
6. The method of claim 5, wherein the first historical rendering frame is one or more of the plurality of historical rendering frames, and the first historical rendering frame is different from the second historical rendering frame; or The rendering time of the first historical rendering frame is earlier than the rendering time of each of the plurality of historical rendering frames; or There are multiple first historical rendering frames, some of which belong to the multiple historical rendering frames, and some of which are different from the multiple historical rendering frames.
7. The method according to claim 5, wherein the time parameter is used to indicate one or more times between the rendering time of the second historical rendering frame and the rendering time of the third rendering frame, wherein the third rendering frame is the next rendering frame to be generated according to a preset frame rate.
8. The method according to claim 5, wherein generating an intermediate frame corresponding to the time parameter based on multiple historical rendering frames and a time parameter comprises: Based on the multiple historical rendering frames, the motion trajectory of the target object is determined. The target object is located in the target scene, and the multiple historical rendering frames are obtained by rendering the target scene. Based on the motion trajectory and the second historical rendering frame, a corresponding intermediate frame is generated.
9. The method according to claim 8, wherein generating the corresponding intermediate frame based on the motion trajectory and the second historical rendering frame comprises: Based on the motion trajectory and the time parameter, the corresponding motion vector is determined, and the time parameter is used to characterize the generation time of the intermediate frame; Based on the motion vector, determine the mapping positions corresponding to multiple pixels in the second historical rendering frame; as well as Based on the mapping positions of the multiple pixels, a corresponding intermediate frame is generated.
10. The method according to claim 1, further comprising: Based on multiple consecutive historical rendering frames, determine the shape change information of the corresponding shadow area; as well as Based on the morphological change information, the shadow area in the target frame is adjusted.
11. The method according to any one of claims 1-10, further comprising: The target scene is rendered using a graphics processor to obtain the plurality of historical rendering frames and / or the first historical rendering frame.
12. A chip, comprising: The first processing circuit is used to generate an intermediate frame corresponding to the time parameters based on multiple historical rendering frames and time parameters. as well as The second processing circuit is used to determine the corresponding target frame based on the first historical rendering frame by repairing the pixel missing areas of the intermediate frame, wherein the target frame is located after the plurality of historical rendering frames.
13. The graphics processor of claim 12, wherein the intermediate frame includes a first intermediate frame and a second intermediate frame, the target frame includes the first target frame, and further comprises: While the floating-point arithmetic unit generates the second intermediate frame, the matrix arithmetic unit determines the corresponding first target frame by repairing the pixel missing areas of the first intermediate frame.
14. The graphics processor of claim 12, wherein the plurality of historical rendering frames includes a second historical rendering frame, the rendering time of the second historical rendering frame being later than the rendering time of the other historical rendering frames in the plurality of historical rendering frames; The first historical rendering frame is one or more of the plurality of historical rendering frames, and the first historical rendering frame is different from the second historical rendering frame; or The rendering time of the first historical rendering frame is earlier than the rendering time of each of the plurality of historical rendering frames; or There are multiple first historical rendering frames, some of which belong to the multiple historical rendering frames, and some of which are different from the multiple historical rendering frames.
15. A server, comprising: Central processing unit (CPU), accelerator, and memory. The CPU is used to acquire multiple historical rendering frames. The accelerator executes instructions in the memory, causing the server to perform the method according to any one of claims 1 to 11.
16. A frame generation apparatus, comprising: The intermediate frame generation unit is configured to generate an intermediate frame corresponding to the time parameters based on multiple historical rendering frames and time parameters. The target frame determination unit is configured to determine the corresponding target frame based on the first historical rendering frame by repairing the pixel missing areas of the intermediate frames, wherein the target frame is located after the plurality of historical rendering frames.
17. The apparatus of claim 16, wherein the intermediate frame comprises a first intermediate frame and a second intermediate frame, and the apparatus further comprises: While generating the second intermediate frame, the corresponding first target frame is determined by repairing the pixel missing areas of the first intermediate frame.
18. The apparatus of claim 16, wherein the plurality of historical rendering frames includes a second historical rendering frame, the rendering time of the second historical rendering frame being later than the rendering time of the other historical rendering frames in the plurality of historical rendering frames; The first historical rendering frame is one or more of the plurality of historical rendering frames, and the first historical rendering frame is different from the second historical rendering frame; or The rendering time of the first historical rendering frame is earlier than the rendering time of each of the plurality of historical rendering frames; or There are multiple first historical rendering frames, some of which belong to the multiple historical rendering frames, and some of which are different from the multiple historical rendering frames.
19. A computer program product comprising machine-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.
