Image processing method, electronic device, and storage medium

Abstract

The present application relates to the technical field of images, and discloses an image processing method, an electronic device, and a storage medium. The method comprises: receiving a first operation of a user; in response to the first operation, starting or exiting a target application; during the starting or exiting, an image compositor determining the number of buffers stacked in a first queue; and when the number of buffers stacked in the first queue is greater than 1, in a first Vsync period, performing composition processing on a first image acquired from a first buffer in the first queue, so as to obtain a first composite image, and upon completing the composition processing on the first image, starting to perform composition processing on a second image acquired from a second buffer in the first queue. The image compositor performs composition processing on the second image in the second buffer in advance without waiting for the wake-up by a Vsync signal, and without waiting for the release of a fence for the display of the previous image frame, thereby prolonging the duration from compositing to sending for display of a buffer, achieving higher fault tolerance of image frame compositing and sending for display, reducing the frame dropping phenomenon, and improving the image display performance.

Description

Image processing methods, electronic devices and storage media Technical Field

[0001] This application relates to the field of image technology, and in particular to an image processing method, electronic device, and storage medium. Background Technology

[0002] The fence mechanism is a resource locking mechanism in the operating system. Its function is to intercept and allow subsequent processes to proceed only when the fence is opened (i.e., released). A fence mechanism also exists in the image frame display process. For example, regarding the display fence, the image compositer (SurfaceFlinger, SF) receives the vertical synchronization signal (Vsync) and is awakened by Vsync. Before compositing the image content (or image data) of the current image frame, it checks whether the display fence of the previous image frame has been released. If the display fence of the previous image frame has been released, then the SF performs compositing processing on the image content of the current image frame.

[0003] When the operating system has insufficient scheduling capabilities or thread abnormalities, the display fence of the image frame is released abnormally, which prevents the SF from being synthesized normally, and thus causes frequent frame drops in electronic devices. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, and a storage medium, which can reduce frame dropping and improve image display performance. The technical solution is as follows:

[0005] Firstly, an image processing method is provided, applied to an electronic device. The method includes: receiving a first operation from a user; and responding to the first operation by launching or exiting a target application. The first operation can be a touch operation, a click operation, a gesture operation, or other operation to launch or exit the application. During the launch or exit of the target application, the method includes: determining the number of buffers stacked in a first queue; if the number of buffers stacked (or piled up) in the first queue is greater than one, within a first Vsync cycle, an image compositer (SF) acquires a first image from the first buffer in the first queue, performs composite processing on the first image to obtain a first composite image, and after completing the composite processing of the first image, within the first Vsync cycle, starts acquiring a second image from the second buffer in the first queue and performs composite processing on the second image. The target application can be any application in a desktop application. Launching the target application refers to the desktop application displaying the interface of the target application corresponding to the application icon after the user triggers the application icon. The first queue is a cache queue corresponding to the images to be composited in the desktop application, and the buffers stacked in the first queue are used to cache the images to be composited obtained after rendering. In other words, SF can proactively consume the second buffer in the first queue of desktop applications, and perform composite processing on the image content (i.e., the second image) in the second buffer in advance. Compared with the native composite process that requires waiting for Vsync to wake up and for the fence to be released, it extends the time from composite to display of the second buffer, leaving more work margin for the composite and display process and improving the fault tolerance of the display.

[0006] Compared to related technologies where the display fence of the previous image frame is not released, and the SF (Synchronous Array) needs to wait for the next Vsync signal before attempting to composite the image content in the second buffer, the embodiment of this application eliminates the need for the SF to wait for the Vsync signal to wake up or for the display fence of the previous image frame to be released. It immediately performs composite processing on the image content in the second buffer, extending the SF's working time, increasing the fault tolerance of image frame composite delivery, reducing frame drops, and improving image display performance. Furthermore, because the SF's working time is extended, events such as delayed delivery or buffer overflows are not recognized, eliminating the need for reporting or notifying, thus reducing the power consumption of the electronic device.

[0007] In one possible implementation, the desktop application's interface triggers tracking events when animations occur, exit animations occur, or during animation states (also known as animation periods). Based on this, the SF (Surface Mount Technology) in the electronic device can identify whether the desktop application's interface displays animations based on tracking events provided by the business system. The business system can be understood as a tracking module with tracking functionality, and the image compositer obtains tracking events through the animation interface provided by the business system. The image compositer identifies the start and completion (or end) of the target application through tracking events, thus obtaining the process of starting the target application; it also identifies the start and completion of the target application's exit through tracking events, thus obtaining the process of exiting the target application. This information is used to subsequently determine whether to enable the pre-compositing scheme to improve image display performance.

[0008] In one possible implementation, the SF (First Component) retrieves the first buffer from the first queue (i.e., the cache queue corresponding to the image to be composited in the desktop application) during the first Vsync cycle, and performs composite processing on the image content (i.e., the first image) in the first buffer to obtain the first composited image. If the desktop application's interface displays animation effects (i.e., the process of starting or exiting the target application), and the number of buffers stacked in the first queue is greater than one, it indicates that the conditions for enabling the pre-compositing scheme are met. After compositing the image content in the first buffer, the SF, without waiting for the wake-up of the Vsync (i.e., the second Vsync message) corresponding to the second buffer, and without waiting for the fence, retrieves the second buffer from the first queue during the first Vsync cycle. The image content (i.e., the second image) in the second buffer is then composited during either the first or second Vsync cycle, effectively forcing the pre-compositing of the second buffer to obtain the second composited image.

[0009] The second Vsync cycle follows the first Vsync cycle, and is the next Vsync cycle after the first Vsync cycle. The first Vsync cycle is the Vsync signal used to wake up the SF to synthesize the first buffer, and its start time is the moment the SF receives the first Vsync message. The second Vsync message is generated after the first Vsync message, and the start time of synthesis of the second buffer is earlier than the moment the SF receives the second Vsync signal. Thus, before the second Vsync message is received, the SF has already started compositing the image content in the second buffer, realizing an early synthesis scheme. Compared with the native synthesis process that requires waiting for Vsync to wake up and for the fence to be released, this extends the SF's working time and improves the fault tolerance of the display. For example, extending the SF's working time increases the workload during the display process if the actual synthesis time of the SF remains unchanged. Furthermore, this early synthesis scheme also increases the workload for situations where the SF may encounter various unexpected events during the actual synthesis process that cause delays in synthesis completion, thereby reducing the probability of frame drops.

[0010] It's important to note that, normally, SF (Synchronous Streaming) synthesizes the second buffer in advance, and both the first and second buffers can be synthesized within the first Vsync cycle. In some cases, the second buffer may not be synthesized in the first Vsync cycle but is synthesized in the second Vsync cycle. The second buffer may be blocked, but it has already received the second Vsync signal, SF has been awakened, and there is already a synthesis event in the queue. The third buffer is waiting to be awakened, so the synthesis of the third buffer will begin immediately after the second buffer is synthesized, and the third buffer can still be synthesized within the second Vsync cycle. In other cases, the synthesis time of the second buffer is particularly long, causing the third buffer to be delayed until the third Vsync signal, but this does not affect the normal process. Moreover, if the second buffer is not synthesized in advance, and its synthesis time is particularly long, the third buffer will drop frames, so synthesizing the second buffer in advance is still meaningful.

[0011] In this embodiment, the condition for enabling the pre-compositing scheme is that the application is in the process of starting or exiting, and the number of buffers stacked in the first queue is greater than 1. The condition for disabling (or ending) the pre-compositing scheme is that the application has completed starting or exiting. When the pre-compositing scheme is enabled, it forces the pre-compositing of one frame without waiting for the fence or the Vsync signal. Then, even if the number of stacked buffers is not greater than 1, the pre-compositing scheme is not exited. Regardless of the number of stacked buffers, the compositing process for each buffer is as follows: the SF waits for the fence, waits for the Vsync signal to wake it up, and then performs compositing processing on the buffer. Due to the serial execution characteristics of the underlying display driver, the composite image also needs to be serially blocked and waited on the display driver side. Therefore, after forcibly pre-compositing one frame (i.e., the second image), even if the number of stacked buffers is still greater than 1, it is not necessary to immediately composite; instead, it waits for the Vsync signal to wake it up before performing compositing processing on the buffer.

[0012] Based on this, after the image content in the second buffer is composited, SF retrieves the third image from the third buffer in the first queue during the second Vsync cycle, performs composite processing on the third image, and obtains the third composite image. SF continues to retrieve the fourth image from the fourth buffer in the first queue during the third Vsync cycle, performs composite processing on the fourth image, and obtains the fourth composite image; this process continues until the target application is launched or exited. In other words, when the desktop application's interface is in animation mode, SF waits for the next Vsync message (corresponding to the second Vsync message) to wake it up, retrieves the third buffer from the first queue, and performs composite processing on the image content (i.e., the third image) in the third buffer to obtain the third composite image; then it waits for the next Vsync message (corresponding to the third Vsync message) to wake it up, retrieves the fourth buffer from the first queue, and performs composite processing on the image content (i.e., the fourth image) in the fourth buffer to obtain the fourth composite image, until the desktop application's interface completes the animation. By enabling the pre-compositing scheme, SF forces the pre-compositing of the second buffer and waits for the next Vsync message (corresponding to the second Vsync message) to wake it up before compositing the image content in the third buffer. In this way, the stacked buffers can be consumed in advance, preventing buffer stacking, and the time from compositing to display of each subsequent image frame is lengthened, resulting in higher fault tolerance and reducing frame dropping.

[0013] In one possible implementation, after the desktop application completes the animation (i.e., completes startup or exit of the target application), the image processing method further includes: if the fence corresponding to the fifth image is detected to be released, the SF performs compositing processing on the fifth image acquired from the fifth buffer of the first queue within the fourth Vsync cycle to obtain the fifth composite image. That is, the image processing method for pre-compositing provided in this application embodiment no longer interferes with the original fence mechanism after the animation is completed, restoring the original fence mechanism and improving the compatibility of the image processing method. The SF waits for the Vsync signal to wake up, and performs compositing processing on the image content in the buffer when the corresponding fence is released.

[0014] In one possible implementation, after determining the number of buffers stacked in the first queue, the image processing method further includes: if the number of buffers stacked in the first queue is 1, and the advance compositing scheme cannot be executed, the SF performs compositing processing on the first image obtained from the first buffer in the first queue within the first Vsync cycle to obtain a first composite image. The SF continues to detect the number of buffers stacked in the first queue, so that if the number of buffers stacked in the first queue is greater than 1, the advance compositing scheme is executed to reduce frame dropping.

[0015] In one possible implementation, when the desktop application's interface displays animation effects and the number of buffers stacked in the first queue is greater than 1, the SF (Show Driver) not only executes the pre-compositing scheme but also updates the offset value used to enable image compositing, obtaining an updated offset value. Based on the second composite image and the updated offset value, the first target image frame is determined, which determines which frame's fence the second composite image should wait for. The fence release time of the first target image frame is earlier than the fence release time of the second target image frame, and the second target image frame is determined based on the second composite image and the offset value. When the display driver releases the display fence of the first target image frame, it sends the second composite image frame for display. This extends the display time of the second composite image during the display process, increases the fault tolerance of the display process, reduces frame drops, and ensures smooth image display, thereby improving image display performance.

[0016] In one possible implementation, the updated offset value enables the display of the second composite image and the N subsequent composite images, where N is a positive integer. Based on the updated offset value and the N subsequent composite images, the SF determines the target image frame for each of the N composite images. The display driver then sequentially displays the N composite images after the fences of their respective target image frames are released. An updated offset value greater than the original offset value extends the display time for both the second composite image and the N subsequent composite images, resulting in higher fault tolerance, reduced frame drops, and smoother display, thus improving image display performance.

[0017] In one possible implementation, SF increments the offset value by one to obtain the updated offset value. This shifts the display fence that the second composite image and the subsequent N composite images would normally wait for under the native mechanism forward by one. The display time for the second composite image and the subsequent N composite images is extended, resulting in higher fault tolerance during the display process, reduced frame drops, smoother visuals, and improved image display performance.

[0018] In one possible implementation, for an offset value of 0, the offset value is incremented by 1 to obtain the updated offset value. Under the native mechanism, it is necessary to wait for the fence of the previous frame of the second image to be released before compositing processing can be performed on the buffer corresponding to the second image (i.e., the second buffer). However, in this embodiment, the updated offset value is 1. When the fence of the two frames preceding the second composite image (i.e., the frame preceding the previous frame of the second composite image) corresponding to the second image is released, the display driver sends the second composite image for display. This process provides higher fault tolerance, reduces frame drops, and ensures smoother image display, thereby improving image display performance.

[0019] For cases where the offset value is 0, the offset value is increased by 2 to obtain the updated offset value. In the native mechanism, it is necessary to wait for the fence of the previous frame of the second image to be released before the second image can be composited. However, in this embodiment, the updated offset value is 2. When the fence of the first three frames (i.e., the frame before the first two frames of the second composite image) corresponding to the second image is released, the display driver sends the second composite image for display. This process provides higher fault tolerance, reduces frame drops, and ensures smoother image display, thereby improving image display performance.

[0020] For an offset value of 1, the offset value is incremented by 1 to obtain the updated offset value. In the native mechanism, it is necessary to wait for the fence of the first two frames of the second image to be released before the buffer corresponding to the second image (i.e., the second buffer) can be composited. However, in this embodiment, the updated offset value is 2. With the fence of the first three frames of the second composite image corresponding to the second image released, the display driver sends the second composite image for display. The process of sending the second composite image has higher fault tolerance, reduces frame drops, and ensures smooth playback, thereby improving image display performance.

[0021] In one possible implementation, after updating the offset value to obtain the updated offset value, SF, upon recognizing the application interface of the desktop application completing the animation effect (i.e., completing the startup or exit of the target application) based on the event points provided by the business system, updates the updated offset value to the original offset value. Thus, the image processing method for pre-synthesis provided in this application embodiment no longer interferes with the original fence mechanism after the animation effect is completed, restoring the original fence mechanism and improving the compatibility of the image processing method.

[0022] In one possible implementation, during the startup or exit of the target application, the desktop application performs rendering processing on the image content and caches the rendered image content in a first queue of buffers. The first queue is the cache queue corresponding to the images to be composited in the desktop application. After enabling the pre-compositing scheme, the image processing method also includes a frame interpolation process in the application thread (i.e., the desktop application). The frame interpolation process is as follows: when the number of buffers stacked in the first queue is less than 1, SF sends a frame interpolation request to the desktop application; in response to the frame interpolation request, the desktop application caches the sixth image rendered in the first queue in the sixth buffer within the current Vsync cycle, and then starts the next rendering in the current Vsync cycle, caching the seventh image rendered in the first queue in the seventh buffer. That is, the desktop application renders the image content in the buffer in advance so that the rendered buffer can be sent to the first queue in a timely manner, reducing frame dropping caused by SF consuming the buffer in advance and improving image display performance.

[0023] The image processing method for pre-compositing provided in this application is applicable to different motion effect scenarios, improving the diversity of applicable scenarios for the pre-compositing scheme. Examples include desktop motion effects, folding motion effects, unfolding motion effects, and focus application motion effects. Desktop motion effects include, but are not limited to: switching from the application icon of a target application in a desktop application to the application interface of the target application (this process is the process of launching the target application), switching from the application interface of the target application to the application icon of the target application in the desktop application (this process is the process of exiting the target application), switching between multiple pages in a desktop application, and switching between multiple applications via the desktop application. Folding motion effects indicate that the display interface of a foldable device switches from an unfolded state to a folded state, and unfolding motion effects indicate that the display interface of a foldable device switches from a folded state to an unfolded state. The display interface is either the interface of the desktop application or the interface of the focus application. Focus application motion effects include, but are not limited to: continuously sliding the application interface of the focus application, causing changes in the interface elements within the application interface of the focus application.

[0024] The first aspect described above introduced the process of launching or exiting the target application in desktop animations. Next, we will explain other animations.

[0025] Secondly, an image processing method is provided, applied to an electronic device. The method includes: receiving a second operation from a user; and, in response to the second operation, switching between multiple pages in a desktop application, or switching between multiple applications via the desktop application. The second operation can be a touch operation, a swipe operation, a gesture operation, etc. During the switching process between multiple pages, or between multiple applications, the number of buffers stacked (or piled up) in a first queue is determined. If the number of buffers stacked in the first queue is greater than one, the method includes: within a first Vsync cycle, an image compositer performs composite processing on a first image obtained from a first buffer in the first queue to obtain a first composite image, and after completing the composite processing on the first image, it begins composite processing on a second image obtained from a second buffer in the first queue. The first queue is a cache queue corresponding to the image to be composited in the desktop application, and the buffers stacked in the first queue are used to cache the image to be composited obtained after rendering. That is, SF can proactively consume the second buffer in the first queue of the desktop application in advance, and perform composite processing on the image content (i.e., the second image) in the second buffer in advance. Compared to the native compositing process, which requires waiting for Vsync to wake up and for the fence to be released, the time from compositing to display of the second buffer is extended, leaving more work margin for the compositing and display process and improving the fault tolerance of display.

[0026] In one possible implementation, during the switching between multiple pages or multiple applications, the desktop application renders the image content and caches the rendered image content in a first queue of buffers. The first queue is the cache queue corresponding to the images to be composited in the desktop application. After enabling the pre-compositing scheme, the image processing method also includes a frame interpolation process in the application thread (i.e., the desktop application). The frame interpolation process is as follows: when the number of buffers stacked in the first queue is less than 1, SF sends a frame interpolation request to the desktop application; in response to the frame interpolation request, the desktop application caches the sixth image rendered in the first queue in the sixth buffer within the current Vsync cycle, and then starts the next rendering in the current Vsync cycle, caching the seventh image rendered in the first queue in the seventh buffer. That is, the desktop application renders the image content in the buffer in advance so that the rendered buffer can be sent to the first queue in a timely manner, reducing frame dropping caused by SF consuming the buffer in advance and improving image display performance.

[0027] Thirdly, an image processing method is provided, applied to an electronic device, which is a foldable device. The method includes: receiving a third operation from a user; and folding or unfolding the foldable device in response to the third operation. During the folding or unfolding of the foldable device, the method includes: determining the number of buffers stacked in a first queue; if the number of buffers stacked in the first queue is greater than 1, within a first Vsync cycle, an image compositer performs composite processing on a first image obtained from a first buffer in the first queue to obtain a first composite image, and after completing the composite processing on the first image, begins composite processing on a second image obtained from a second buffer in the first queue. The first queue is a cache queue corresponding to the image to be composited in the desktop application, and the buffers stacked in the first queue are used to cache the image to be composited obtained after rendering. That is, SF can proactively consume the second buffer in the first queue of the desktop application in advance, and perform composite processing on the image content (i.e., the second image) in the second buffer in advance. Compared to the native compositing process, which requires waiting for Vsync to wake up and for the fence to be released, the time from compositing to display of the second buffer is extended, leaving more work margin for the compositing and display process and improving the fault tolerance of display.

[0028] In one possible implementation, the step of folding or unfolding the foldable device in response to the third operation can be exemplified by: switching the display interface of the foldable device from an unfolded state to a folded state in response to the third operation, or switching the display interface of the foldable device from a folded state to an unfolded state in response to the third operation. The display interface is the interface corresponding to the desktop application or the interface corresponding to the focused application.

[0029] In one possible implementation, during the folding or unfolding of the foldable device, the desktop application or the focus application performs rendering processing on the image content and caches the rendered image content in a first queue of buffers; the first queue is the cache queue corresponding to the image to be composited in the desktop application or the focus application. After enabling the pre-compositing scheme, the image processing method also includes a frame interpolation process in the application thread (i.e., the desktop application or the focus application). The frame interpolation process is as follows: when the number of buffers stacked in the first queue is less than 1, the image compositor sends a frame interpolation request to the desktop application or the focus application; in response to the frame interpolation request, the desktop application or the focus application caches the rendered sixth image in the sixth buffer of the first queue within the current Vsync cycle, and then starts the next rendering within the current Vsync cycle, caching the rendered seventh image in the seventh buffer of the first queue. That is, the desktop application or the focus application renders the image content in the buffer in advance so as to send the rendered buffer to the first queue in a timely manner, reducing frame dropping caused by SF consuming the buffer in advance and improving image display performance.

[0030] Fourthly, an image processing method is provided, applied to an electronic device. The method includes: receiving a fourth operation from a user; responding to the fourth operation by continuously sliding the application interface of a focused application, or by a change in the interface elements of the application interface of the focused application; wherein the fourth operation can be a touch operation, a swipe operation, a click operation, a gesture operation, etc. During the continuous sliding of the application interface of the focused application, or during a change in the interface elements of the application interface of the focused application, the method includes: determining the number of buffers stacked in a first queue; if the number of buffers stacked (or piled) in the first queue is greater than 1, within a first Vsync cycle, the image synthesizer performs synthesis processing on a first image obtained from the first buffer in the first queue to obtain a first synthesized image, and after completing the synthesis processing of the first image, it begins to perform synthesis processing on a second image obtained from the second buffer in the first queue. The first queue is a buffer queue corresponding to the image to be synthesized in the focused application, and the buffers stacked in the first queue are used to cache the image to be synthesized obtained after rendering. That is, SF can proactively consume the second buffer in advance for the second buffer in the first queue of the focused application, and perform synthesis processing on the image content (i.e., the second image) in the second buffer in advance. Compared to the native compositing process, which requires waiting for Vsync to wake up and for the fence to be released, the time from compositing to display of the second buffer is extended, leaving more work margin for the compositing and display process and improving the fault tolerance of display.

[0031] In one possible implementation, during the continuous scrolling of the application interface of the focused application, or during changes in the interface elements of the focused application, the focused application performs rendering processing on the image content and caches the rendered image content in a first queue of buffers; the first queue is the cache queue corresponding to the images to be composited in the focused application. After enabling the pre-compositing scheme, the image processing method also includes a frame interpolation process in the application thread (i.e., the focused application). The frame interpolation process is as follows: when the number of buffers stacked in the first queue is less than 1, the image compositor sends a frame interpolation request to the focused application; in response to the frame interpolation request, the focused application caches the sixth image rendered in the first queue of the sixth buffer in the current Vsync cycle, and then starts the next rendering in the current Vsync cycle, caching the seventh image rendered in the first queue of the seventh buffer. That is, the focused application renders the image content in the buffer in advance so as to send the rendered buffer to the first queue in a timely manner, reducing frame dropping caused by SF consuming the buffer in advance and improving image display performance.

[0032] The image processing procedures (including the timing of ending the pre-compositing scheme, the updating and restoration of offset values) and technical effects under other motion effects in the second, third and fourth aspects mentioned above can be found in the technical means corresponding to the various embodiments of starting or exiting motion effects in the first aspect mentioned above, and will not be repeated here.

[0033] This application provides an image processing method applied to an electronic device. The method includes: determining whether the electronic device is in a preset scene; if the electronic device is in a preset scene, determining the number of buffers stacked in a first queue, wherein the preset scene includes a target application startup scene, a target application exit scene, a scene of switching between multiple pages in a desktop application of the electronic device, a scene of switching between multiple applications via a desktop application, a folding scene when the electronic device is a foldable device, an unfolding scene when the electronic device is a foldable device, a continuous sliding scene of the application interface of the focus application, a scene of changing interface elements in the application interface of the focus application, and other animation scenes; the buffers stacked in the first queue are used to cache the image to be composited after rendering; if the number of buffers stacked in the first queue is greater than 1, within a first Vsync cycle, the SF performs composite processing on the first image obtained from the first buffer in the first queue, and after completing the composite processing of the first image, starts to perform composite processing on the second image obtained from the second buffer in the first queue; if the number of buffers stacked in the first queue is 1, within a first Vsync cycle, the SF performs composite processing on the first image obtained from the first buffer in the first queue.

[0034] Fifthly, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image processing method as described in the first aspect or any implementation thereof.

[0035] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the image processing method involved in the first aspect or any implementation thereof.

[0036] In a seventh aspect, embodiments of this application provide a chip system that can be applied to an electronic device. The chip system includes one or more processors that invoke computer instructions to cause the electronic device to perform the image processing method as described in the first aspect or any implementation thereof.

[0037] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the image processing method described in the first aspect or any implementation thereof.

[0038] The technical effects achieved by the fifth, sixth, seventh and eighth aspects mentioned above are similar to the technical effects achieved by the corresponding technical means in the first to fourth aspects mentioned above, and will not be repeated here. Attached Figure Description

[0039] Figure 1 is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0040] Figure 2 is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0041] Figure 3 is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0042] Figure 4 is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0043] Figure 5 is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0044] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0045] Figure 7 is a block diagram of a software system for an electronic device provided in an embodiment of this application;

[0046] Figure 8 is a schematic diagram of the process from drawing and rendering to display of an image frame provided in an embodiment of this application;

[0047] Figure 9 is a schematic diagram of the buffer flow process during image frame display provided in an embodiment of this application;

[0048] Figure 10 is a schematic diagram of a GPU task fence provided in an embodiment of this application;

[0049] Figure 11 is a schematic diagram of a fence provided in an embodiment of this application;

[0050] Figure 12 is a schematic diagram of a waiting fence provided in an embodiment of this application;

[0051] Figure 13 is a schematic diagram of an image synthesis provided in an embodiment of this application;

[0052] Figure 14 is a schematic diagram of a frame dropping phenomenon provided in an embodiment of this application;

[0053] Figure 15 is a schematic diagram of SF advance synthesis tracking provided in an embodiment of this application;

[0054] Figure 16 shows a flowchart of SF advance synthesis provided in an embodiment of this application;

[0055] Figure 17 is a flowchart of another SF advance synthesis provided in an embodiment of this application;

[0056] Figure 18 is a schematic diagram of an application of frame interpolation provided in an embodiment of this application;

[0057] Figure 19 is a schematic diagram of another frame loss problem tracking provided by an embodiment of this application;

[0058] Figure 20 is a flowchart of an image processing method provided in an embodiment of this application;

[0059] Figure 21 is a flowchart of the interaction between threads in an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0061] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, user interface, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0063] The image processing method provided in this application embodiment is applicable to scenarios where the application interface of a desktop application or a focused application in an electronic device is in a motion effect state, that is, scenarios where the application interface has enabled continuous frame refresh operation. Continuous frame refresh refers to continuously drawing, rendering, compositing, and displaying multiple consecutive image frames, which form an animation. The motion effect scenario can be one or more of desktop motion effects, collapse / expand motion effects, and focused application motion effects. The motion effect scenario is described below.

[0064] Desktop animations refer to the use of animations, transitions, and other techniques to enhance the user's interaction with the interface within the desktop environment. Desktop animations include, but are not limited to, launch animations, unlock animations, exit animations, side-swiping animations, scrolling animations, icon scaling, and other animations that interact with the desktop. Desktop animations include at least one of the following: switching from the target application's icon in the desktop application to the target application's interface (this process is the launch of the target application); switching from the target application's interface to the target application's icon in the desktop application (this process is the exit of the target application); switching between multiple pages in the desktop application; and switching between multiple applications via the desktop application.

[0065] For example, when a user opens (i.e., launches) or closes (i.e., exits) a target application (also known as a target application) on a desktop application of an electronic device, switches between multiple pages in the desktop application, or switches between multiple applications through the desktop application, there will be a smooth animation transition effect. With animation effects, such as the pop-up or disappearance of windows, the blurring or sharpening of the background, etc., the operation process is more natural and smooth, and the user's visual experience is improved.

[0066] Electronic devices can have a split-screen function, allowing users to switch between multiple applications via a desktop application; multiple applications can run in the background, and users can access the task manager on the desktop and switch between them via the desktop application.

[0067] It should be noted that the search application does not have an application icon. Users can switch to the application interface of the search application by long swiping or diagonal swiping, which is a kind of opening (i.e., launching) of the application.

[0068] Multitasking view functionality in electronic devices allows users to view thumbnails or windows of multiple applications simultaneously. Multitasking animations include, but are not limited to, switching between different applications. For example, when multiple tasks are running on an electronic device, multitasking animations can be used to demonstrate the switching process when a user switches from one application to another, such as scaling, rotating, or panning application windows, allowing the user to intuitively see the application currently being switched.

[0069] In some embodiments, when the electronic device is a foldable device (e.g., a foldable phone), the content on the screen transitions smoothly with animation when the foldable phone is folded or unfolded. When the foldable phone is folded or unfolded, the display interface on the foldable phone's screen can be the interface corresponding to a desktop application or the interface corresponding to a focused application. The folding animation indicates that the foldable device's display interface switches from an unfolded state to a folded state, and the unfolding animation indicates that the foldable device's display interface switches from a folded state to an unfolded state, wherein the display interface is the interface corresponding to a desktop application or the interface corresponding to a focused application.

[0070] In some embodiments, the focus application animation includes continuously sliding the application interface of the focus application, and changes in the interface elements within the application interface of the focus application.

[0071] The focus application animation effect can be a sliding animation effect, that is, continuously sliding to display content within the application interface of the focus application. A focus application refers to a pre-defined application (e.g., a whitelist), where content within the application interface is continuously slid to display the focus application animation effect.

[0072] The focus application animation can be a transition animation or a launch animation. Interface elements can be files or controls, with files organized as folders, favorites, etc., and controls organized as menus, settings options, or subpages. Changes in the interface elements of the focus application can refer to transition or launch animations occurring on files (e.g., images or documents) or controls (e.g., options or buttons) within the focus application.

[0073] The image processing method for pre-compositing provided in this application is applicable to different motion effect scenarios, improving the diversity of applicable scenarios for the pre-compositing scheme. Several examples illustrate these motion effect scenarios below.

[0074] Scene 1: Desktop Animation

[0075] For example, as shown in Figure 1, which is a schematic diagram of an application scenario of an image processing method provided in this application embodiment, taking a mobile phone as the electronic device and a camera application as the target application, the mobile phone desktop includes a camera application icon. Figure 1 shows the desktop animation effect of launching the camera application. Three representative image frames are shown. It can be understood that the desktop animation effect also includes other image frames. The process of launching the camera application to take pictures is as follows: The user clicks (also known as touch or press, etc.) the camera application icon on the mobile phone desktop, as shown in Figure A in Figure 1. The desktop animation effect starts, and the application interface of the desktop application starts a continuous frame refresh operation; next, the mobile phone will display the screen in the preview interface of the camera application. When the screen appears in the preview interface, it will go through a slow appearance process (e.g., zooming) as shown in Figure B in Figure 1, until it reaches Figure C in Figure 1. At this point, the desktop animation effect ends (or the animation effect is completed).

[0076] The animation effects in this application embodiment also include animation effects from the application interface to the corresponding application icon. For example, the reverse process (or inverse process) in Figure 1 above, that is, the animation effect from the screen in the preview interface to the camera application icon. Desktop animation effects may include animation effects for launching the target application and animation effects for exiting the target application. The animation effect for launching the target application can be represented as the process shown in Figure 1 above, and the animation effect for exiting the target application can be represented as the reverse process (or inverse process) in Figure 1 above.

[0077] Scene 2, Sliding Animation 1

[0078] For example, as shown in Figure 2, which is a schematic diagram of an application scenario of an image processing method provided in this application embodiment, taking a mobile phone as an example, Figure 2 shows the desktop animation when switching between multiple pages on the mobile phone desktop, with three representative image frames shown. It can be understood that the desktop animation also includes other image frames. The switching process between multiple pages on the mobile phone desktop is as follows: The user starts to slide on the current page in the sliding direction (for example, to the left), as shown in Figure A of Figure 2. The desktop animation starts, and the application interface of the desktop application starts a continuous frame refresh operation; the current page slides out along the sliding direction, and the next page slides in along the sliding direction. After the switching process shown in Figure B of Figure 2, until Figure C of Figure 2, the desktop animation ends.

[0079] Scene 3, Sliding Animation 2

[0080] The sliding animation effect includes not only the animation effect generated by sliding left and right in Scene 2, but also the animation effect generated by sliding in other directions, such as the animation effect generated by sliding up and down or sliding down to the left in Figure 3 below.

[0081] For example, as shown in Figure 3, which is a schematic diagram of an application scenario of an image processing method provided in this application embodiment, taking a mobile phone as an example, Figure 3 shows the desktop animation when switching to the application interface of the search application, with three representative image frames. It can be understood that the desktop animation also includes other image frames. The process of opening the search application is as follows: The user starts to slide on the desktop in the sliding direction. The sliding duration is longer than the sliding duration when switching between multiple pages on the mobile phone desktop in Figure 2A above, which can be called a long swipe operation, as shown in Figure 3A. The desktop animation starts, and the application interface of the desktop application starts a continuous frame refresh operation; then the application interface of the search application will appear on the mobile phone. When the application interface of the search application appears, it will go through the appearance process shown in Figure 3B (for example, blurring) until it reaches Figure 3C. At this point, the desktop animation ends.

[0082] Scene 4: Focus Application Animation 1

[0083] For example, as shown in Figure 4, which is a schematic diagram of an application scenario of an image processing method provided in this application embodiment, taking a mobile phone as the electronic device, a photo album as the focus application, and images as the file as an example, Figure 4 shows the focus application animation corresponding to changes in interface elements in the application interface, illustrated by three representative image frames. It can be understood that the focus application animation also includes other image frames. The process of image changes in the application interface is as follows: When the user clicks (or touches) the folder corresponding to the camera image, as shown in Figure A of Figure 4, the animation starts, and the application interface of the focus application starts a continuous frame refresh operation; next, the images in the folder corresponding to the camera image can be expanded like a book, realizing a transition animation. When multiple images in the folder are expanded, they will go through a slow appearance process (e.g., zooming) as shown in Figure B of Figure 4, until Figure C of Figure 4, at which point the desktop animation ends.

[0084] Understandably, for controls in the focus application that use menus, settings options, or subpages as storage, when the user clicks on the corresponding control, the menu, settings options, or subpages can also expand like a book, presenting the transition animation shown in Figure 4.

[0085] Scene 5, Focus Application Animation 2

[0086] The featured application in Figure 4 above and Figure 5 below can be a browser, reading application, chat application, shopping application, telephone application, etc. Figure 4 above uses the photo album as an example of the featured application, and Figure 5 below uses the browser as an example of the featured application.

[0087] For example, as shown in Figure 5, which is a schematic diagram of an application scenario of an image processing method provided in this application embodiment, taking a mobile phone as the electronic device and a pre-set browser as the focus application as an example, Figure 5 shows the focus application animation effect corresponding to continuous sliding in the browser's application interface, illustrated by two representative image frames. It can be understood that the focus application animation effect also includes other image frames. The continuous sliding display process in the application interface is as follows: The user starts sliding in the sliding direction on the current interface of the browser, as shown in Figure A of Figure 5. The animation effect starts, and the application interface of the focus application starts continuous frame refreshing operation; the current interface slides out along the sliding direction, and the next interface slides in along the sliding direction until Figure B of Figure 5 is reached, at which point the animation effect ends. Of course, after Figure B of Figure 5, continuous sliding can continue to start the next round of animation effect.

[0088] It should be noted that the documents in Figure 5A and Figure 5B can also be replaced with images.

[0089] In this embodiment, the user performs continuous frame refresh operations by swiping up, down, left, right, or diagonally. When the electronic device receives a notification from the application interface that continuous frame refresh has been enabled, it performs frame drawing, rendering, and compositing processes based on the continuous frame refresh operation to display the content corresponding to the continuous frame refresh operation.

[0090] This application's embodiments involve a display fence mechanism. When the SF (Synchronous Frame Array) composites the image content of the current image frame (i.e., the image content in the buffer corresponding to the current image frame), it also checks whether the display fence of the previous image frame has been released. If the display fence of the previous image frame has been released, the SF composites the image content of the current image frame; if the display fence of the previous image frame has not been released, the SF directly abandons (or skips) the compositing process of the image content of the current image frame, waiting for the next Vsync to wake it up, and then tries to composite the image content of the current image frame. Under this fence mechanism, in cases where the operating system has insufficient scheduling capabilities or thread abnormalities, the display fence of the image frame may be released abnormally, which may lead to frequent frame drops on the electronic device. When continuous frame refresh is enabled in the application interface, this frame drop phenomenon will manifest as stuttering and discontinuity, affecting the user experience.

[0091] Based on the aforementioned technical problem of frame dropping, this application provides an image processing method for pre-compositing images, applied in scenarios where electronic devices are in motion effect mode. Taking desktop motion effect as an example, the method receives a first operation from the user; in response to the first operation, it starts or exits a target application, wherein the first operation can be a start operation or an exit operation; during the process of starting or exiting the target application, the method includes: determining the number of buffers stacked in a first queue; if the number of buffers stacked (or piled up) in the first queue is greater than 1, within a first Vsync cycle, SF obtains a first image from the first buffer in the first queue, performs composite processing on the first image to obtain a first composite image, and immediately starts obtaining a second image from the second buffer in the first queue within the first Vsync cycle, and performs composite processing on the second image. The target application can be any application in the desktop application. Starting the target application refers to the desktop application displaying the interface of the target application corresponding to the application icon after the user triggers the application icon of the target application. The first queue is a cache queue corresponding to the image to be composited in the desktop application, and the buffers stacked in the first queue are used to cache the image to be composited obtained after rendering. In other words, SF (Frame Buffer) can proactively consume the second buffer in the first queue of desktop applications without waiting for the Vsync signal to wake it up or for the fence of the previous image frame to be released. This allows for pre-processing of the image content (i.e., the second image) within the second buffer. Compared to the native compositing process, which requires waiting for Vsync to wake it up and for the fence to be released, this extends the time from compositing to display of the second buffer, providing more leeway for the compositing and display process and improving display fault tolerance. Furthermore, because the SF's working time is extended, events such as delayed display or buffer overflows will not be recognized, eliminating the need for reporting or notification, thus reducing the power consumption of electronic devices. In this embodiment, the buffer can be understood as a frame buffer.

[0092] The electronic devices involved in the embodiments of this application are described herein. These electronic devices include, but are not limited to, mobile phones, foldable phones, smart wearable devices, portable Android devices (PADs), desktop computers, laptops, handheld computers, netbooks, laptops, desktop computers, portable computers, PDAs, wireless terminal devices, smart home devices, communication devices, Global Positioning System (GPS) navigators, digital broadcast receivers, personal digital assistants (PDAs), portable media players (PMPs), ultra-mobile personal computers (UMPCs), augmented reality (AR) devices, virtual reality (VR) devices, and other electronic devices including those with touchscreens. The embodiments of this application do not limit the specific type of the electronic device.

[0093] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Referring to Figure 6, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0094] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0096] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0097] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0098] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0099] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0100] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0101] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0102] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0103] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0104] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0105] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0106] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D and application processor.

[0107] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0108] The software system of electronic device 100 will be described next. The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system of electronic device 100.

[0109] Figure 7 is a block diagram of a software system of an electronic device 100 provided in an embodiment of this application. Referring to Figure 7, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer (Framework), the hardware abstraction layer (HAL), and the kernel layer (kernel).

[0110] The application layer can include a series of application packages. As shown in Figure 7, application packages can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and desktop.

[0111] The application layer includes the application (APP) thread, which is used to be woken up by the Vsync signal to perform drawing and rendering.

[0112] The application layer also includes a business system, which is used to mark events for desktop animations, focused application animations, and collapsing animations, and provides animation interfaces to the image compositing system in the application framework layer. For desktop and collapsing animations, the animation interface can be a desktop interface; for focused application animations, the animation interface can be an application interface. The image compositing system can identify whether the desktop application or the focused application is in an animation state based on the event points obtained from the animation interface.

[0113] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 7, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0114] The application framework layer may also include an image compositing system. The image compositing system controls image compositing and generates Vsync signals. The image compositing system includes an image compositor (SF), a Vsync thread, and a queue buffer thread. The SF is awakened by the Vsync signal to perform compositing processing. The Vsync thread generates the next Vsync signal based on Vsync signal requests. The queue buffer thread stores buffered data, generates Vsync signal requests, and wakes up the SF, etc. The image compositor (SF) can also be called the compositing thread.

[0115] SF (Synchronizer) is the service responsible for compositing screen display content. It receives image buffers from multiple applications and system services, composites them onto the screen, and forms the final display effect. SF includes a listener (DispSyncSource.cpp). SF can also be a native layer located between the application framework layer and the HAL (Hardware Algorithm) layer.

[0116] Desktop animations and focus application animations may be supported by basic services or components provided by the application framework layer. Therefore, the business system used to generate desktop animations and focus application animations can also be located in the application framework layer.

[0117] The hardware abstraction layer can include a hardware compositor (HWComposer, HWC), which can also be called a hardware blending renderer. HWC can be used to composite image content.

[0118] The kernel layer is the layer between hardware and software, and can also be called the driver layer. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0119] Hardware can include displays, audio devices, Bluetooth devices, camera devices, sensor devices, etc.

[0120] The kernel layer is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operating system's operation, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel can determine the timing of an application's operations on certain parts of the hardware.

[0121] The kernel layer includes hardware-dependent programs such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules such as clock management and process scheduling modules, and critical data structures. The kernel layer can be located in the processor or embedded in internal memory.

[0122] In one possible implementation, the application thread in the application layer draws and renders the image frame, and then the application thread sends the drawn and rendered image frame to the buffer queue of SF. Whenever the Vsync signal arrives, SF retrieves the image frames from the buffer queue in sequence for compositing.

[0123] For ease of understanding, the examples provided are for reference only and are related to the concepts in the embodiments of this application.

[0124] 1. Frame: Refers to a single frame, the smallest unit of visual information displayed on a screen. A frame can be understood as a still image; rapidly displaying multiple consecutive frames can create the illusion of motion. Frame rate refers to the number of frames refreshed per second, or the number of times the graphics processor in an electronic device refreshes the screen per second. A higher frame rate results in smoother and more realistic animation. The more frames per second, the smoother the displayed motion.

[0125] It should be noted that before the interface displays a frame, it usually needs to go through processes such as drawing, rendering, and compositing.

[0126] 2. Frame rendering: This refers to the rendering of images on the display interface. The display interface can consist of one or more views, each of which can be drawn by the visual controls of the view system. Each view is composed of subviews, and a subview corresponds to a small component within the view. For example, one subview might correspond to a symbol in the image view.

[0127] 3. Frame rendering: This involves coloring the drawn view or adding 3D effects. For example, 3D effects can include lighting effects, shadow effects, and texture effects.

[0128] 4. Frame Compositing: This is the process of combining multiple rendered views into a display interface.

[0129] Next, referring to Figures 8-14, we will introduce the image frame display process, the buffer flow process and fence mechanism during the image frame sending process, and the frame dropping phenomenon.

[0130] The Vsync signal is used to synchronize the display's refresh rate and the GPU's rendering rate. It can be generated by hardware and instructs the system to refresh the display for optimal display performance. The refresh rate indicates the number of times the screen is refreshed per second. Vsync signals are divided into hardware Vsync signals and software Vsync signals. Hardware Vsync signals are abbreviated as Vsync-hw signals, while software Vsync signals include applied vertical sync (Vsync-app) signals and synthesized vertical sync (Vsync-sf) signals. For ease of description, Vsync-hw, Vsync-app, and Vsync-sf signals are simplified to Vsync-hw, Vsync-app, and Vsync-sf, respectively.

[0131] Vsync-app and Vsync-sf are generated based on Vsync-hw. For example, Vsync-hw is virtualized into Vsync-app and Vsync-sf by a listener in SF (DispSyncSource.cpp). Vsync-hw can be generated by the display driver based on the refresh rate. DispSyncSource.cpp samples Vsync-hw, and the sampling result is input into the software Vsync model. DispSyncSource.cpp runs the software Vsync model to output Vsync-app and Vsync-sf.

[0132] The electronic device utilizes Vsync-app, Vsync-sf, and Vsync-hw to draw, render, and display image frames, as shown in Figure 8. Figure 8 is a schematic diagram illustrating the process of drawing, rendering, and displaying image frames according to an embodiment of this application. In Figure 8, the image frames displayed by the electronic device are sequentially frame 1, frame 2, and frame 3. For example, taking the display of frame 1, under the action of the Vsync-app signal, the application (APP) thread of the electronic device draws and renders frame 1 through the view system of the application framework layer. This process is the APP drawing and rendering process, and the rendered image content can be stored in a buffer. After the drawing and rendering of frame 1 is completed, the application thread of the electronic device sends the completed frame 1 to the SF in the image compositing system. Under the action of Vsync-sf, the SF in the image compositing system reads the image content from the buffer and sends the image content to HWC, which then performs compositing processing on the completed frame 1. After compositing is complete, the electronic device starts the display driver by calling the kernel layer. Under the action of Vsync-hw, the display driver displays the content corresponding to frame 1 on the screen (i.e., the display screen). This process is called HWC display delivery. After the screen finishes displaying frame 1, the display driver calls the kernel thread (crtc_commit) to release the fence. Releasing the fence indicates that the display of one image frame is complete. Frames 2 and 3 are also composed and displayed in a similar process to frame 1, which will not be described in detail here.

[0133] As shown in Figure 8, an image frame goes through three cycles from drawing, rendering to display. In these three cycles, the first cycle completes the APP drawing and rendering, the second cycle completes the SF compositing, and the third cycle completes the HWC display. The periods of the three signals Vsync-app, Vsync-sf, and Vsync-hw can be determined according to the refresh rate of the APP. For example, if the refresh rate of the APP is 120Hz, the period of the three signals Vsync-app, Vsync-sf, and Vsync-hw is 8.3 milliseconds (ms); if the refresh rate of the APP is 90Hz, the period of the three signals Vsync-app, Vsync-sf, and Vsync-hw is 11.1 milliseconds (ms); and if the refresh rate of the APP is 60Hz, the period of the three signals Vsync-app, Vsync-sf, and Vsync-hw is 16.6 ms.

[0134] For example, as shown in Figure 9, which is a schematic diagram of the buffer flow process during image frame display according to an embodiment of this application. Figure 9 uses a mobile phone as an example to illustrate the buffer rotation process in the current mobile phone system. As shown in Figure A of Figure 9, after the App receives Vsync-app, it starts filling or drawing the image content of the current image frame (i.e., the image content in the buffer corresponding to the current image frame) through the User Interface (UI) thread. Then, it passes the drawing instructions to the RenderThread thread through the native API interface and wakes up the RenderThred component. The RenderThread thread processes the drawing instructions and passes them to the GPU for rendering the current image frame. If the RenderThread thread's queue buffer is full, it needs to wait for the buffer to be dequeued, that is, wait for the rendering of the previous image frame to be completed before rendering the image content of the current image frame when there is an available buffer. After rendering is completed, the number of buffers in the blast buffer queue (BBQ) is increased by 1. The BBQ is used to store the image content. The rendered buffer is transferred to SF via an asynchronous binder, and a transaction is set to prepare for image frame compositing. The number of buffers in SF's buffer queue is incremented by one. Under Vsync-sf, SF consumes the buffer, that is, it composites the image content of the previous image frame in the buffer. After compositing, it is sent to the display driver. Under Vsync-hw, the display driver sends the image frame to the display screen. After the image frame is displayed, SF releases the buffer. BBQ receives the release buffer from SF, and the number of buffers in BBQ is decremented by one. RenderThred has available buffers, ends the dequeue buffer wait, and can execute the rendering process for the current image frame. In other words, SF maintains buffers for all layers and, at the correct time, overlays the buffers that should be displayed and sends them to the screen for display. After the image content of the current image frame is fully displayed, SF will send a callback message to each application thread to release the buffer, making the buffer available and drawable for the application thread. At this point, the complete cycle of a buffer is completed.

[0135] In this embodiment, the number of buffers refers to the number of buffers containing image content. The number of buffers is illustrated in Figure 9B. BBQ acts as the buffer queue for SF. Assume BBQ originally had two buffers, buffer1 and buffer2, waiting to be composited. After the application thread finishes rendering, it sends the buffers to BBQ for queuing, such as buffer3 in BBQ, which means the number of buffers in BBQ increases by 1, resulting in 3 buffers waiting to be composited in BBQ. SF takes the image content from buffer1 for composite processing and sends buffer1 to the display driver's buffer queue for queuing. The display driver's buffer queue originally had one buffer, buffer0, waiting to be displayed, with buffer1 placed after buffer0. The display driver displays buffer0. After displaying buffer0, SF releases buffer0 (not shown in Figure 9B). Then, the display driver displays buffer1. After displaying buffer1, SF releases buffer1. At this point, BBQ still has two buffers, buffer2 and buffer3, waiting to be combined, which means the number of buffers in BBQ is reduced by 1. The buffer1 released by SF is called the available buffer, which refers to a buffer that does not contain image content. This available buffer can be transferred to the application thread.

[0136] It should be noted that Figure 9 illustrates the flow of a single buffer. Each stage (including rendering, compositing, and display submission) can involve two, three, or more buffers in turn. In Figure 9, on the App side, each vertical line represents the rendering of an image frame, and each vertical line includes multiple sub-steps in the rendering process (represented by multiple superimposed boxes). On the SF side, each vertical line represents the compositing of an image frame, and each vertical line includes multiple sub-steps in the compositing process (represented by multiple superimposed boxes). On the display driver side, each vertical line represents the display submission of an image frame, and each vertical line includes multiple sub-steps in the display submission process (represented by multiple superimposed boxes).

[0137] In this embodiment of the application, various fence mechanisms exist during the drawing, rendering, and display of image frames. For example, the fence in a graphics processing unit (GPU), as shown in Figure 10, is a schematic diagram of a GPU task fence provided in this embodiment of the application. Figure 10 shows the display driver in the kernel layer waiting for the release of the rendering task's fence. When a rendering task is sent to the GPU, a fence is registered. Only after the rendering task's fence is released is the image frame allowed to be displayed on the screen. For example, in Figure 10, before time t0, the display driver waits for the rendering task's fence to be released. At time t0, the display driver waits for the rendering task's fence to be released and begins displaying.

[0138] The fence mechanism involved in this application embodiment is a visible fence mechanism, as shown in Figure 11. Figure 11 is a schematic diagram of a visible fence provided by this application embodiment, illustrating the trace process of the visible fence of an image frame. In this application embodiment, the Vsync signal that wakes up the SF is Vsync-sf, which will be referred to as Vsync in the following simplified description. Figure A in Figure 11 shows that when Vsync wakes up the SF to composite the image content of the current image frame (i.e., the image content in the buffer corresponding to the current image frame), it is detected that the visible fence of the previous image frame has not been released. The SF directly abandons (or skips) the compositing process of the image content of the current image frame and waits for the next Vsync to wake it up, resulting in frame dropping. Figure B in Figure 11 shows a schematic diagram of waiting for the fence to be released. When Vsync wakes up the SF to composite the image content of the current image frame, it is detected that the visible fence of the previous image frame has been released, and the SF performs the compositing of the image content of the current image frame.

[0139] Based on the above introduction to the fence mechanism, the following explains the principle behind setting the display fence. The calculation process for the display fence that the SF (Surfing System) needs to wait for requires the SF's work-duration. The SF's work-duration mechanism refers to the minimum time interval from when the SF starts compositing the image content of the image frame to when the image frame is displayed. As shown in Figure 12, which is a schematic diagram of a fence waiting mechanism provided in an embodiment of this application, Figure 12 illustrates the SF's work-duration mechanism. Assuming the SF's work-duration is 15.6ms, that is, the image frame currently composed by the SF will not be displayed on the screen until at least 15.6ms later.

[0140] The display fence that SF waits for comes from the driver. SF will only begin compositing the current image frame after the underlying driver releases the display fence for a given frame. Whether SF waits for the display fence of the previous frame (also called the preceding image frame) or the display fence of two previous frames (also called the two preceding image frames) is determined by the refresh rate of the current application. Here, "two previous frames" refers to the frame before the previous frame. The image frame corresponding to the display fence that SF waits for is determined by an offset value (shift), where shift = static_cast.<size_t> The function is defined as ((mExpectedPresentTime-now) / VsyncPeriod), where now represents the time when the image content of the current frame is composed, mExpectedPresentTime is the expected display time of the current frame, and (mExpectedPresentTime-now) represents the SF work-duration, i.e., the duration between the start of SF composition (now) and the expected display time (mExpectedPresentTime). VsyncPeriod refers to the current Vsync period, representing the duration between the start of composition of frame N and the start of composition of frame N+1. The current Vsync period is related to the current refresh rate (i.e., the refresh rate corresponding to the app). The SF work-duration can be pre-configured in the operating system and can be configured differently according to the refresh rate corresponding to the app. For example, if the current refresh rate is 120Hz, the current Vsync period is 8.3ms, and the SF work-duration can be set to 10.3ms; if the current refresh rate is 60Hz, the current Vsync period is 16.6ms, and the SF work-duration can be set to 15.6ms.

[0141] `shift` indicates the frame number of the previous frame to wait for; that is, the `shift` value indicates which frame before the current frame the frame corresponds to. `static_cast`<size_t> () is the rounding function. The shift value is the integer part of (mExpectedPresentTime-now) divided by the current Vsync cycle, which is the integer part of the SF work-duration divided by the current Vsync cycle.

[0142] For example, if the current refresh rate is 60Hz and the current Vsync period is 16.6ms, assuming the current work-duration is set to 15.6ms, then shift = 0 (rounded down from 15.6 / 16.6). This means it waits for the fence of the previous image frame before the current image frame. In other words, the prerequisite for SF to composite the image content of the current image frame is waiting for the fence of the previous image frame to be released. This implies that the Nth frame waits for the fence of the (N-1)th frame, where N is a positive integer. In other words, the prerequisite for SF to composite the image content of the current image frame is waiting for the fence of the previous image frame to be released. If the current refresh rate is 120Hz and the current Vsync period is 8.3ms, assuming the current work-duration is set to 10.3ms, then shift = 1 (10.3 / 8.3 rounded down). This means that the frame waiting for the fences of the two preceding frames is required for SF to composite the image content of the current frame. In other words, SF's prerequisite for compositing the image content of the current frame is waiting for the fences of the two preceding frames to be released. This implies that frame N cannot wait for the fence of frame N-1; it needs to wait for the fence of frame N-2, where N is a positive integer. In other words, the prerequisite for SF to composite the image content of the current frame is waiting for the fences of the two preceding frames to be released.

[0143] Assume that before compositing the image content of the current image frame, the SF (Synchronizer) needs to check whether the display fence of the i-th frame preceding the current image frame has been released, where i is a positive integer. The SF will only perform image compositing on the current image frame if the display driver calls the kernel thread `crtc_commit` to release the display fence of the i-th frame preceding the current image frame. In other words, when the SF is woken up by Vsync to perform image compositing on the current image frame, it checks whether the display fence of the i-th frame preceding the current image frame has been released. If the fence has not been released, the SF waits for a preset time (e.g., 1ms, 2ms, etc.). If the fence is still not released, the SF skips or abandons the compositing of the current image frame's image content. That is, if the SF does not wait for the display fence of the i-th frame preceding the current image frame to be released, the SF will not perform image compositing on the current image frame. It waits for the next Vsync wake-up call before attempting to composit the image content of the current image frame again.

[0144] Based on the above method of calculating the offset value (shift), the image frames that the current image frame needs to wait for are shown in Figure 12. The cases of i=1 and i=2 are explained. Figure 12 shows the frame dropping problem caused by the fence release that cannot wait for the display of the previous image frame and the fence release that is not waiting for the display of the previous two frames.

[0145] Figure A in Figure 12 shows the fence waiting for the display of the previous image frame. Since the fence for the display of the previous image frame is released only after the SF's Vsync, Vsync wakes up the SF and waits for the fence for the display of the previous image frame. This can be done by waiting for a preset time (e.g., 1ms, 2ms, etc.) and finding that the fence for the display of the previous image frame has not been released, then the current image frame returns directly.

[0146] As shown in Figure 12, at point ①, the SF receives Vsync and is woken up by it. Since the display fence of the previous image frame (also known as the preceding frame) has been released (not shown in Figure 12, A), the SF can perform image synthesis of the first image frame (i.e., the image content in the buffer corresponding to the first image frame) at point ①. At point ② (i.e., time t0 shown in Figure 12, A), the SF receives Vsync and is woken up by it. The SF detects that the display fence of the first image frame it is waiting for has not been released. The SF directly skips or abandons the synthesis of the image content of the second image frame (i.e., the image content in the buffer corresponding to the second image frame), enters sleep mode, and waits for the next wake-up to attempt to synthesize the image content of the second image frame. This results in frame dropping (the synthesis of the image content of the second image frame should have been performed at point ②). That is, at time t0 shown in Figure 12, the display fence of the first image frame has not been released, resulting in frame dropping. Subsequently, at point ③, since the fence for displaying the first image frame has been released, at point ④, SF receives Vsync, is awakened by Vsync, and begins to synthesize the image content of the second image frame.

[0147] Figure 12B illustrates the fence for the display of the previous two frames. Since the fence for the display of the previous two frames is released only after the Vsync of the SF, the Vsync wakes up the SF and waits for the fence for the display of the previous two frames. It can wait for a preset time (e.g., 1ms, 2ms, etc.) and find that the fence for the display of the previous two frames has not been released, so the current image frame returns directly.

[0148] As shown in Figure 12B, at point ①, the SF receives and is woken up by Vsync. Since the display fences for the first two frames of the first image frame have been released (not shown in Figure 12B), the SF can perform image content composition for the first image frame at point ①. The display driver sends the first image frame to the display at point ②. At point ③, the SF receives and is woken up by Vsync. The SF detects that the display fences for the first two frames of the second image frame it is waiting for have been released. Therefore, the SF begins image content composition for the second image frame at point ③. At point ④ (i.e., time t0 as shown in Figure B of Figure 12), the SF receives Vsync and is woken up by Vsync. The SF detects that the display fences of the first two frames (i.e., the first image frame) of the third image frame that it is waiting for have not been released. The display fence of the first image frame is only released at point ⑤. Therefore, the SF directly skips or abandons the compositing process of the image content of the third image frame (i.e., the image content in the buffer corresponding to the third image frame), enters sleep mode, and waits for the next wake-up to try to compose the image content of the third image frame. This results in frame dropping (the compositing of the image content of the third image frame should have been performed at point ④). In other words, there is a long running time in the process of sending the first image frame for display, so that the display fence of the first image frame has not been released at time t0 as shown in Figure B of Figure 12, and is only released after the display is completed at point ⑤, resulting in frame dropping.

[0149] For example, as shown in Figure 13, which is a schematic diagram of image compositing provided in an embodiment of this application, Figure 13 takes an APP with a refresh rate of 60Hz, a current Vsync period of 16.6ms, and a current work-duration of 15.6ms as an example. It can be calculated that shift = 0, that is, the prerequisite for SF to compose an image frame is to wait for the display fence of the previous image frame of the image frame to be released. The significance of the above display fence setting will be explained below with reference to Figure 13. As shown in Figure 13, at ①, SF receives Vsync and is woken up by Vsync. Since the display fence of the previous image frame of the first image frame has been released, SF can compose the image content of the first image frame at ①. Normally, at point ② (i.e., time t0 shown in Figure 13), the SF receives and is woken up by Vsync. The display fence of the first image frame has been released, and the SF can composite the image content of the second image frame. However, due to insufficient system resources (software / hardware resources) causing system scheduling abnormalities, or abnormalities in the APP's threads, it may be necessary to wait for the GPU task's fence to be released, thus prolonging the display time of the first image frame. As a result, the fence, which should have been released before point ②, is still not released at point ②. Thus, at point ②, the SF receives and is woken up by Vsync. Before the SF can composite the image content of the second image frame, it detects that the display fence of the previous image frame (i.e., the first image frame) has not been released. After waiting for a preset time (e.g., 1ms, which can be started from the wake-up time), the display fence of the first image frame is still not released, and the SF will not perform image content composite processing for the second image frame. In this scenario, the SF will skip or abandon the compositing process for the second image frame. The second image frame returns, and the SF enters sleep mode, waiting to be woken up again to attempt to compose the image content of the second image frame. This results in frame dropping at point ②, specifically at time t0 shown in Figure 13, when the display fence for the first image frame is not released, leading to frame dropping. Subsequently, 16.6ms later, the SF at point ③ receives Vsync and is woken up. Since the display fence for the first image frame has been released, the SF can begin compositing the image content of the second image frame at point ③, and the composed second image frame is then displayed at point ④.

[0150] As shown in the image frame composition process in Figure 13, the significance of setting the fence is that even if the SF is forcibly awakened by Vsync at point ② for composition processing, due to the serial execution characteristics of the underlying display driver, the second image frame will still wait on the underlying display driver side, and the second image frame will still be displayed at point ④. This is the same as the display effect of compositing when awakened at point ③. In other words, whether the SF is awakened by Vsync at point ② for the composition of the second image frame or the SF is awakened by Vsync at point ③ for the composition of the second image frame, both result in the image being displayed at point ④. Therefore, when calculating shift, the ratio of work-duration to the current Vsync is used. Although frame dropping may occur, it can meet the display requirements of the image frames. That is, if the current image frame (corresponding to the second image frame) cannot wait for the fence (corresponding to the first image frame) to be released, there is no difference in display between compositing in this frame (corresponding to point ③) and compositing in the next image frame (corresponding to point ④). However, since SF cannot wait for the fence of the previous image frame to be released, it skips or abandons the synthesis of the image content of the current image frame, which will cause the frame dropping phenomenon at point ②.

[0151] It should be noted that in this embodiment of the application, the determination of whether a frame is dropped is based on whether the underlying display driver runs for a long time, rather than on the frame rate (SF). In Figure 13, a frame is dropped during the second image frame display process, that is, a blank frame occurs, resulting in display stuttering.

[0152] Based on the description of the fence mechanism in Figures 11-13 above, it can be seen that if the SF (Frame Freeze) cannot wait for the fence to be released, thus skipping or abandoning the composition of the image content of the current image frame, it will cause frame drops. If these frame drops occur in motion effects, they are very easily perceived by the user, causing a stuttering feeling and affecting the user experience. The usual solution to frame drops is to increase the frequency of the relevant chip. For example, for frame drops caused by long CPU running times, increasing the CPU frequency can reduce running time and thus reduce frame drops. However, simply increasing the frequency cannot solve all stuttering and frame drop problems.

[0153] For example, as shown in Figure 14, Figure A in Figure 14 illustrates the frame dropping phenomenon caused by the inability to increase the frequency under high-temperature thermal limiting and low-battery modes. Figure A in Figure 14 uses the fence waiting for the display of the previous two frames as an example. Due to a long runtime issue in the process of sending frame 1 to the display driver, the fence release time for frame 1 is later than the time the SF receives the Vsync. Therefore, when frame 3 should be being synthesized, the synthesis of frame 3 is skipped because the fence for frame 1 has not been released. Only after being woken up by the next Vsync does the SF begin synthesizing frame 3. When the display driver should be displaying frame 3, because frame 3 has not yet been synthesized, the display driver continues to display the image frame before frame 3 (i.e., frame 2). This is indicated by a dashed box in Figure 14, resulting in frame dropping and display stuttering. For frame dropping caused by thermal limiting and low battery, increasing the frequency will lead to high power consumption and high heat generation. Therefore, when electronic devices (e.g., mobile phones) reach the upper limit of temperature safety (e.g., above 46°C), or when users actively turn on power saving mode to save power, the problem of frame dropping cannot be avoided by increasing the frequency. Users will feel that the game is lagging when they encounter frame dropping.

[0154] For example, as shown in Figure 14, Figure B in Figure 14 illustrates frame dropping caused by a sudden long running time in a certain frame of SF. The load of electronic device systems (e.g., mobile phones) is instantaneous and unpredictable, and the running time of various functions can vary. For example, due to system-wide issues, a certain thread may suddenly have a long running time in a certain frame, leading to frame dropping and display stuttering. In this case, increasing the CPU frequency may be meaningless and will not solve the frame dropping problem. In Figure B of Figure 14, because SF has a long running time in the process of composing frame 2, the display driver continues to display the image frame before frame 2 (i.e., frame 1) when it should be displaying frame 2, since frame 2 has not yet been completed. Figure B in Figure 14 is represented by a dashed box, indicating frame dropping and display stuttering.

[0155] For example, in multi-threaded operation, there are some locking states. Sometimes, threads need to wait for locks, that is, wait for scheduling. When electronic devices (e.g., terminal devices) are under heavy load, the CPU is running at almost full frequency, and the phenomenon of threads not being scheduled or waiting for locks often occurs. If such problems occur on critical display threads, frame drops will occur, causing stuttering.

[0156] Based on the frame loss phenomenon mentioned above, the SF request synthesis scheme and application layer frame interpolation scheme provided in the embodiments of this application will be introduced next with reference to Figures 15-19.

[0157] As illustrated in Figure 9 above, SF maintains buffers for all layers and overlays the buffers that should be displayed at the correct time, sending them to the screen for display. In other words, SF has a buffer queue for all layers, so SF can sense how many buffers have accumulated for each application waiting to be consumed (i.e., the image content in the buffer corresponding to the image frame needs to be composited). These accumulated buffers waiting to be consumed are not discarded; they are taken from the buffer queue frame by frame by SF for composited according to the production order.

[0158] An image frame goes through drawing, rendering, and display. For ease of distinction, in this embodiment, the image content in the buffer during the drawing / rendering process and the compositing process is referred to as an image; the image content in the buffer after compositing and awaiting display is referred to as a composite image; and the image content in the buffer during the display process is referred to as an image frame. It should be noted that Figures 8-14 above are all described primarily using image frames, without distinguishing between images, composite images, and image frames.

[0159] The operating system of electronic devices supports the rendering of animation effects. Business systems in electronic devices can monitor preset operation events (such as swiping the screen, clicking a button, touching the screen, folding the screen, etc.), determine whether to enable animation effects according to the preset rules of the operating system, and mark the points when the animation effect is displayed on the application interface of the desktop application or the focused application, and provide animation effect interfaces (desktop interface or application interface) for the point events related to the animation effect. SF can recognize the animation effect occurring on the application interface of the desktop application or the focused application through this animation effect interface, that is, recognize that the application interface has started continuous frame refresh operation.

[0160] In this embodiment, the application interface of a desktop application or a focused application triggers a tracking event when an animation effect occurs, an exit animation effect occurs, or the application is in an animation state (also known as an animation period). Based on this, after a user operates the electronic device, the SF in the electronic device can identify whether the application interface of the desktop application or the focused application displays an animation effect based on the tracking events provided by the business system. For example, for desktop application animation effects, the SF identifies the start and completion (or end) of the target application in the desktop application based on the tracking events provided by the business system, thereby obtaining the process of starting the target application; or, it identifies the start and completion (or end) of the exit of the target application in the desktop application, thereby obtaining the process of exiting the target application; or, it identifies the start and completion (or end) of multiple page switching, thereby obtaining the process of multiple page switching; or, it identifies the start and completion (or end) of multiple application switching, thereby obtaining the process of multiple application switching. For folding animation effects, when the SF identifies the display interface of the foldable device as the interface corresponding to the desktop application or the focused application based on the tracking events provided by the business system, the foldable device starts folding and... The folding process of the foldable device is obtained by completing (or ending) the folding animation. For the unfolding animation, SF identifies the foldable device's display interface as either a desktop application or the interface corresponding to the focus application based on the event points provided by the business system, thus obtaining the unfolding process of the foldable device. For the focus application animation, SF identifies the application interface of the focus application from the start of continuous sliding to the completion of continuous sliding (or the end of continuous sliding) based on the event points provided by the business system, thus obtaining the process of continuously sliding the application interface of the focus application. Alternatively, it identifies the interface elements in the application interface of the focus application from the start of change to the completion of change (or the end of change), thus obtaining the process of change of interface elements in the application interface of the focus application. This is to determine whether to enable the pre-compositing scheme to improve image display performance.

[0161] Based on this, this application provides an image processing method for pre-compositing. During the startup or exit of the target application, the SF determines the number of buffers stacked in the first queue. Within the first Vsync cycle, the SF retrieves a first image from the first buffer in the first queue and performs compositing processing on the first image to obtain a first composite image. If the number of buffers stacked (or piled up) in the first queue is greater than 1, it indicates that the conditions for enabling the pre-compositing scheme are met. After completing the compositing processing of the first image, without waiting for the wake-up of the Vsync (i.e., the second Vsync message) corresponding to the second buffer, and without waiting for the fence, the SF starts retrieving a second image from the second buffer in the first queue within the first Vsync cycle and performs compositing processing on the second image. Completing the compositing processing of the second image within the first Vsync cycle or the second Vsync cycle is equivalent to forcibly pre-compositing the second image in the second buffer to obtain a second composite image.

[0162] In this process, the second Vsync cycle follows the first Vsync cycle, and is the next Vsync cycle after the first Vsync cycle. The first Vsync cycle is the Vsync signal used to wake up the SF (Synchronous Streaming Center) to synthesize the first buffer, and its start time is the moment the SF receives the first Vsync message. The synthesis of the second buffer begins before the SF receives the second Vsync message corresponding to the second buffer, and the second Vsync message is generated after the first Vsync message. Since the synthesis of the second buffer begins earlier than the moment the SF receives the second Vsync signal, the SF has already begun synthesizing the image content (i.e., the second image) in the second buffer before receiving the second Vsync message, achieving an early synthesis scheme. Compared to the native synthesis process that requires waiting for Vsync to wake up and for the fence to be released, this extends the SF's working time, improves the display fault tolerance, and thus reduces the probability of frame drops.

[0163] In some embodiments, after the image content in the second buffer is composited, the SF retrieves the third image from the third buffer in the first queue during the second Vsync cycle, performs composite processing on the third image, and obtains the third composite image; the SF continues to retrieve the fourth image from the fourth buffer in the first queue during the third Vsync cycle, performs composite processing on the fourth image, and obtains the fourth composite image; and so on, until the target application is launched or exited. That is, when the application interface of the desktop application or the focused application is in an animation state, the SF waits for the next Vsync message (corresponding to the second Vsync message) to wake it up, retrieves the third buffer from the first queue, and performs composite processing on the image content (i.e., the third image) in the third buffer to obtain the third composite image; then it waits for the next Vsync message (corresponding to the third Vsync message) to wake it up, retrieves the fourth buffer from the first queue, and performs composite processing on the image content (i.e., the fourth image) in the fourth buffer to obtain the fourth composite image, until the application interface of the desktop application or the focused application completes the animation. By enabling the pre-compositing scheme, SF forces the pre-compositing of the second buffer and waits for the next Vsync message (corresponding to the second Vsync message) to wake it up before compositing the image content in the third buffer. In this way, the stacked buffers can be consumed in advance, preventing buffer stacking, and the time from compositing to display of each subsequent image frame is lengthened, resulting in higher fault tolerance and reducing frame dropping.

[0164] In one possible implementation, after the desktop application or the focused application completes the animation effect, the image processing method further includes: if the fence corresponding to the fifth image is detected to be released, the SF performs compositing processing on the fifth image acquired from the fifth buffer of the first queue within the fourth Vsync cycle to obtain the fifth composite image. That is, the image processing method for pre-compositing provided in this application embodiment no longer interferes with the original fence mechanism after the animation effect is completed, restoring the original fence mechanism and improving the compatibility of the image processing method. The SF waits for the Vsync signal to wake up, and performs compositing processing on the image content in the buffer when the corresponding fence is released.

[0165] In one possible implementation, after determining the number of buffers stacked in the first queue, the image processing method further includes: if the number of buffers stacked in the first queue is 1, and the advance compositing scheme cannot be executed, the SF performs compositing processing on the first image obtained from the first buffer in the first queue within the first Vsync cycle to obtain a first composite image. The SF continues to detect the number of buffers stacked in the first queue, so that if the number of buffers stacked in the first queue is greater than 1, the advance compositing scheme is executed to reduce frame dropping.

[0166] In this embodiment, the conditions for enabling the pre-compositing scheme are that the application interface of the desktop application or the focused application displays animation effects and the number of buffers stacked in the first queue is greater than 1. The conditions for disabling (or ending) the pre-compositing scheme are that the application interface of the desktop application or the focused application completes the animation effects. When the pre-compositing scheme is enabled, it forcibly pre-composites one frame without waiting for the fence or the Vsync signal. Then, even if the number of stacked buffers is not greater than 1, it does not exit the pre-compositing scheme. Regardless of the number of stacked buffers, the compositing process for each buffer is as follows: the first frame does not wait for the fence, waits for the Vsync signal to wake up, and performs compositing processing on the buffer. The pre-compositing scheme only ends after the animation effects are completed.

[0167] This application proposes a scheme for proactive pre-compositing in SF (Screen Rendering). When SF detects that the application interface of a desktop application or the focused application is in a motion effect state, if SF senses that there are two or more buffers waiting to be consumed in the first queue, then when SF completes the compositing task of the previous image of the current image, it immediately performs compositing processing on the current image without waiting for the current image's Vsync to wake up or for the fence. It removes the buffers waiting to be consumed stored in the first queue for pre-compositing, which is equivalent to forcibly pre-compositing the current image. At the same time, in this frame and thereafter, all fences that SF would have to wait for under the native mechanism are moved forward by one, to avoid the frame being unable to be passed to the hardware layer (i.e., the display driver) due to the underlying fence not being released.

[0168] In this embodiment, after the SF (Synthetic Rendering) completes the synthesis of the second image, it also needs to send the synthesized second image to the display driver for display on the screen. When the application interface of a desktop application or a focused application displays animation effects, and the number of buffers stacked in the first queue is greater than 1, the SF not only executes the pre-synthesis scheme but also updates the offset value used to enable image synthesis, obtaining an updated offset value. Based on the second synthesized image and the updated offset value, the first target image frame is determined, that is, the frame whose fence the second synthesized image must wait for is determined. The first target image frame is displayed before the second target image frame determined based on the second synthesized image and the offset value. When the fence of the first target image frame is released, the display driver sends the second synthesized image for display. Because the fence release time of the first target image frame is earlier than the fence release time of the second target image frame, the display time of the second synthesized image during the display process is extended, the fault tolerance of the display process is higher, frame dropping is reduced, the screen is smooth, and thus the image display performance is improved.

[0169] It should be noted that, under the native mechanism, the image content in the buffer is composited only after the fence is detected being released. In the image processing method provided in this application embodiment, since the image is composited before the fence is released, the composited buffer is sent for display upon detection of the fence being released. In the image processing method provided in this application embodiment, the fence serves to enable the display of the composited buffer.

[0170] Figure 13 above has already explained the significance of the display fence setting. In this embodiment, when SF detects that the application interface of the desktop application or the focused application is in a dynamic state, and there are currently two or more buffers waiting to be consumed in the first queue, SF actively initiates the pre-compositing scheme. The calculated fence to be waited for is then updated, that is, the offset value (shift) is updated. For example, the offset value is incremented by one to obtain the updated offset value. This is equivalent to pre-compositing the current image and the N images following it, regardless of the fence.

[0171] By implementing the SF proactive pre-synthesis scheme provided in this application embodiment, the working time of SF can be extended, the fault tolerance of image frame synthesis and display can be improved, frame dropping can be reduced, and the screen can be smooth, thereby improving image display performance. As shown in Figure 15, Figure 15 is a schematic diagram of SF pre-synthesis tracking provided in this application embodiment. Figure 15 shows the SF tracking diagram when SF pre-synthesizes. When the application interface of the desktop application or the focus application is detected to be in an animation state, if SF senses that there are two or more buffers in the first queue, when SF completes the synthesis task of the previous image of the current image (i.e., frame 1 shown in Figure 15), it does not wait for the Vsync wake-up of the current image (i.e., frame 2 shown in Figure 15) and does not wait for the fence, and immediately performs synthesis processing on the current image (i.e., frame 2 shown in Figure 15), which is equivalent to forcibly pre-synthesizing the current image (i.e., pre-synthesizing frame 2), and the number of buffers changes from 2 to 1. In other words, assuming frame 1 is the current image, when the SF completes the compositing task for the current image, it immediately performs compositing processing on the next image without waiting for the Vsync wake-up or the fence. For images after the current image, such as frame 3, the SF waits for the Vsync wake-up and performs compositing processing on frame 3 without waiting for the fence. Similarly, for images after frame 3, the SF also waits for the Vsync wake-up and begins compositing processing without waiting for the fence. The composited image is then sent to the display driver for display, as shown in Figure 15 for frames 1, 2, and 3.

[0172] Simultaneously, when the desktop application or the focused application's interface is detected to be in motion, if SF detects that the first queue currently contains two or more buffers, it updates the offset value shift to obtain an updated offset value shift'. The updated offset value shift' enables the display of the second composite image and the N composite images following it. N is a positive integer, and the value of N is related to the duration of the motion effect; the later the motion effect ends, the larger the value of N, and the earlier the motion effect ends, the smaller the value of N. Based on the updated offset value shift' and the N composite images following the second composite image, SF determines the target image frame for each of the N composite images. With the fence of the target image frame of each of the N composite images released, the display driver sequentially displays the N composite images. The updated offset value shift' is greater than the offset value shift, which extends the display duration of the second composite image and the N composite images following it, resulting in higher fault tolerance in the display process, reduced frame drops, smoother screen display, and improved image display performance.

[0173] It should be noted that, in the native mechanism, the offset value shift is used to enable image content to undergo compositing. In the image processing method provided in this application embodiment, updating the offset value shift' enables the already composited image content (i.e., the composite image) to be displayed.

[0174] For example, assuming a refresh rate of 60Hz, a current Vsync cycle of 16.6ms, and a current work-duration of 15.6ms, the offset value shift = 0 can be calculated. If SF detects that the first queue currently contains two or more buffers, it modifies the shift value, for example, by adding m to shift, where m is a positive integer, resulting in the updated offset value shift' = shift + m.

[0175] Due to the serial execution characteristics of the underlying display driver, image compositing on the underlying display driver side still involves waiting. In other words, the display delivery process is a serial blocking waiting process. Even if shift' = shift + 2, moving the display fence that needs to be waited for under the native mechanism forward by two, the underlying display driver side still needs to wait serially after image compositing. Therefore, moving the display fence that needs to be waited for by one (i.e., m = 1) forward by two (i.e., m = 2) achieves the same effect as moving it forward by two (i.e., m = 2). In other words, adding 1 or 2 to shift achieves the same effect.

[0176] Similarly, modifying the shift value by adding 3, 4, etc., has the same effect as adding 1 to the shift value.

[0177] Based on this, in this embodiment, the offset value shift is incremented by one to obtain the updated offset value shift' = shift + 1. Thus, the display fence that the current composite image (i.e., the second composite image) and the N composite images following it would normally wait for under the original mechanism is shifted forward by one. The display time for the current composite image and the N subsequent composite images is extended, resulting in higher fault tolerance during the display process, reduced frame drops, and smoother image display, thereby improving image display performance.

[0178] In some embodiments, for an offset value of 0, the offset value is incremented by 1 to obtain the updated offset value. Under the native mechanism, it is necessary to wait for the fence of the previous frame of the second image to be released before the buffer corresponding to the second image (i.e., the second buffer) can be composited. However, in this embodiment, the updated offset value is 1. When the fence of the two frames preceding the second composite image (i.e., the frame preceding the previous frame of the second composite image) is released, the display driver sends the second composite image for display. This process provides higher fault tolerance, reduces frame drops, and ensures smoother image display, thereby improving image display performance.

[0179] For offset values ​​of 0, the offset value is increased by 2 to obtain the updated offset value. In the native mechanism, it is necessary to wait for the fence of the previous frame of the second image to be released before the image content in the second buffer can be composited. However, in this embodiment, the updated offset value is 2. When the fence of the first three frames (i.e., the frame before the first two frames of the second composite image) corresponding to the second image is released, the display driver sends the second composite image for display. This process provides higher fault tolerance, reduces frame drops, and ensures smoother image display, thereby improving image display performance.

[0180] For an offset value of 1, the offset value is incremented by 1 to obtain the updated offset value. In the native mechanism, it is necessary to wait for the fence of the first two frames of the second image to be released before the image content in the second buffer can be composited. However, in this embodiment, the updated offset value is 2. With the fence of the first three frames of the second composite image released, the display driver sends the second composite image for display. The process of sending the second composite image has higher fault tolerance, reduces frame drops, and ensures smooth playback, thereby improving image display performance.

[0181] For example, when shift=0, under the native fence mechanism, if the fence for displaying frame N-1 is released late, frame N will not be able to wait for the fence for displaying frame N-1. SF skips or abandons the compositing of the image content of frame N, and does not perform compositing at the time when frame N should be compositing. Instead, it composites frame N at the time when frame N+1 should be compositing, resulting in frame dropping. With the image processing method provided in this application embodiment, shift'=shift+1=0+1=1. When sending frame N for display, it waits for the fence for displaying frame N-2. After the image content of frame N-1 is composed, the image content of frame N is immediately processed for compositing. After the fence for displaying frame N-2 is released, the display driver sends frame N for display, thus extending the time from compositing to sending frame N.

[0182] For example, taking the Nth frame as the current image, the SF (Synthetic Fraction) redetermines the target image frame corresponding to the fence that the current synthesized image needs to wait for to be displayed based on the shift' value. For example, shift = 0, m = 1, shift' = shift + m = 0 + 1 = 1, and the target image frame is the two frames before the current synthesized image. Under the native fence mechanism, the SF performs synthesis processing on the image content of the current image (i.e., the image content in the buffer corresponding to the current image) while waiting for the fence of the previous image frame to be displayed to be released. However, in this embodiment, the SF performs synthesis processing on the current image after the synthesis of the previous image frame is completed, and sends the current synthesized image for display while waiting for the fence of the two frames before the current synthesized image to be displayed to be released. This extends the time from synthesis to display of the Nth frame, improves the fault tolerance of the display, and reduces the probability of frame dropping.

[0183] In the native mechanism, the compositing process occurs once for each falling or rising edge in the first queue. In Figure 15, when the buffer count is 2, the pre-compositing scheme is executed. If there are no falling or rising edges, the SF immediately performs pre-compositing on the current image. Once compositing is complete, the buffer count drops to 1. In other words, in Figure 15, when the SF detects that the desktop or focused application interface is in motion and the buffer count is 2, it immediately performs compositing on the current image while completing the compositing task of the previous image frame. The buffer count then becomes 1, effectively inserting a frame during the compositing process.

[0184] According to the native fence mechanism, the current frame (corresponding to the current image) will return directly because it cannot wait for the fence, and wait for the next wake-up of the SF. In this embodiment, the current frame and all frames after it are pre-synthesized, and the fence of the current frame and all frames after it is shifted forward by one, that is, the shift value is modified by incrementing the calculated shift value by 1. By modifying the shift value, the work-duration of the SF corresponding to the current frame and each subsequent frame is lengthened, allowing more time for system scheduling exception recovery, thread exception recovery, etc., resulting in higher fault tolerance and making it less likely for frame drops to occur. Furthermore, because the work-duration of the SF is lengthened, events such as delayed display or buffer overload will not be recognized, thus eliminating the need for tracking or reporting. It can also slow down the system scheduling frequency, for example, by appropriately reducing the operating frequency of the processing chip, which can be a central processing unit (CPU) or a graphics processing unit (GPU), thereby reducing the power consumption of the electronic device.

[0185] In this embodiment, although the current image and the N subsequent images are pre-combined, the actual display process cannot be advanced; it still requires waiting for the previous image frame to be displayed. This appears to extend the waiting time for each frame to be displayed, but this is not frame dropping. During SF composition, the time from composition to display for each frame is also extended, but this is not frame dropping. Furthermore, by extending the time from composition to display for each frame, the probability of frame dropping due to delays in composition caused by unforeseen circumstances is reduced.

[0186] The extended work-duration effect of SF can be illustrated by Figure 13 above. Figure 13 describes the image frame as the main body, without distinguishing between the image, the composite image, and the image frame. Figure 13 shows that SF needs to wait for the display fence of the previous image frame to be released before SF can composite an image frame. In the native mechanism, at point ②, SF is woken up by Vsync. Before SF can composite the image content of the second image frame, SF detects that the display fence of the previous image frame (i.e., the first image frame) has not been released. SF directly skips or abandons the composite processing of the image content of the second image frame. Then, 16.6ms later, at point ③, SF is woken up by Vsync. Since the display fence of the first image frame has been released, SF can start compositing the image content of the second image frame at point ③. However, the embodiment of this application performs composite processing of the image content of the current image frame in advance, that is, immediately after the image content of the first image frame is composited, the image content of the second image frame is composited.

[0187] Compared to Figure 13, where the SF is activated at point ③ to perform image content compositing on the second image frame, the work-duration of the SF in the image processing scheme for requesting compositing provided in this embodiment can be extended. For example, in Figure 13, the SF is activated at point ③ to perform image content compositing on the second image frame, and the SF's work-duration is 15.6ms. However, when executing the image processing scheme for requesting compositing provided in this embodiment, the image content of the second image frame is composited immediately after the image content of the first image frame is completed, and the SF's work-duration is greater than 15.6ms + 16.6ms = 32.2ms. Since the SF performs image content compositing on the second image frame immediately after the image content of the first image frame is completed, compared to the native mechanism where the image content of the second image frame is composited at point ③, the SF has more time to perform the compositing of the image content of the second image frame.

[0188] The technical effect of the extended work-duration of SF can be illustrated by FIG. 16. As shown in FIG. 16, FIG. 16 is a flowchart of an SF pre-synthesis provided by an embodiment of the present application. FIG. 16 describes the main body as image frames, without distinguishing between images, synthesized images, and image frames. In FIG. 16, the white square represents the Nth frame of pre-synthesis, and the Nth frame can correspond to frame 2 in FIG. 15 above. The solid line represents the SF synthesis and display sequence under the native mechanism. That is, under the native mechanism, the third black square should be displayed at the position of the white square in the display, and then the black squares are synthesized and displayed in sequence according to the solid line. When the pre-synthesis scheme is enabled, the white square is sent for display at the moment corresponding to the white square according to the dotted line, occupying the original position of the third black square, and the subsequent black squares are shifted and displayed in sequence according to the dotted line. In FIG. 16, assuming that the work-duration of SF is 11 ms and the current Vsync period is 8 ms, calculate shift = 1 (rounded value of 11 / 8). Under the native mechanism, SF synthesizes each image frame in sequence and sends each image frame to the display driver for display according to the solid arrow line. When the pre-synthesis scheme is enabled, SF actively pre-synthesizes the Nth frame and sends each image frame to the display driver for display according to the dotted arrow line. Explaining the work-duration of the Nth frame from the time axis, the work-duration of the N-2th frame and the N-1th frame (that is, the first two black squares) are both 11 ms, and the working duration is relatively short; the work-duration of the Nth frame (that is, the white square) is 11 ms < t < 19 ms, and the work-duration of the image frames after the Nth frame (that is, the black squares behind the white square) are all 11 ms + 8 ms = 19 ms. Thus, through the pre-synthesis scheme, SF has more abundant time to execute the synthesis of the image content of each frame, extends the duration from synthesis to display of each frame, leaves more time for system scheduling exception recovery, thread exception recovery, etc., has higher fault tolerance, and reduces the probability of frame loss caused by delays in synthesis due to unexpected situations during the synthesis process.

[0189] It should be noted that FIG. 16 shows that after the pre-synthesis of the white square, then, under the wake-up of the Vsync signal, the black squares after the white square are synthesized. The synthesis process of the subsequent black squares is: wait for the wake-up of the Vsync signal for synthesis. Even so, due to not waiting for the fence, the duration from synthesis to display of each frame is still extended.

[0190] In some embodiments, after updating the offset value to obtain the updated offset value, in one implementation, the SF can continuously use the updated offset value to pass the composite image to the display driver for display. Even if the pre-compositing scheme is exited, the updated offset value is still used to perform the compositing and display process, without interfering with the native fence mechanism, thus achieving compatibility with the native fence mechanism and improving the applicability of the pre-compositing scheme. In another implementation, when the SF recognizes that the application interface of the desktop application or the focused application has completed the animation effect based on the point events provided by the business system, it updates the updated offset value to the original offset value. In this way, the image processing method for pre-compositing provided in this application embodiment no longer interferes with the native fence mechanism after the animation effect is completed, restoring the native fence mechanism.

[0191] In some embodiments, during the startup or exit of the target application, the desktop application performs rendering processing on the image content and caches the rendered image content in a first queue of buffers. The first queue is the cache queue corresponding to the image to be composited in the desktop application. After enabling the pre-compositing scheme, the image processing method also includes a frame interpolation process in the application thread (i.e., the desktop application). The frame interpolation process is as follows: when the number of buffers stacked in the first queue is less than 1, the SF sends a frame interpolation request to the desktop application; in response to the frame interpolation request, the desktop application caches the sixth image rendered in the first queue in the sixth buffer within the current Vsync cycle, and then starts the next rendering in the current Vsync cycle, caching the seventh image rendered in the first queue in the seventh buffer. That is, the desktop application renders the image content in the buffer in advance so as to send the rendered buffer to the first queue in a timely manner, reducing frame dropping caused by the SF consuming the buffer in advance and improving image display performance.

[0192] As shown in Figure 8, this image processing method is applied to an electronic device, which includes a surface simulation (SF) and an application thread (i.e., a desktop application). The application thread performs the rendering process before the SF performs the compositing process; that is, the application thread is above the SF. When the application thread renders an image, it needs to wait for the application vertical synchronization (Vsync-app) signal to wake it up. After rendering the previous image (i.e., the sixth image) of the current image is completed, it waits for Vsync-app to wake up before rendering the current image (i.e., the seventh image). This embodiment of the application executes an advance compositing scheme. The SF advances the compositing of the buffers in the first queue. When it detects that the number of buffers stacked in the first queue is less than 1, it sends a frame insertion request to the application thread. The frame insertion request can be Vsync-app, causing the application thread to render the image content of the current image (i.e., the seventh image) in advance. According to the frame interpolation request, after the application thread finishes drawing and rendering the previous image (i.e., the sixth image) in the current Vsync cycle, it immediately starts drawing and rendering the current image (i.e., the seventh image) in the current Vsync cycle. After the drawing and rendering of the seventh image is completed, the drawn and rendered seventh image is cached in the seventh buffer in the first queue.

[0193] For example, in response to a frame interpolation request, if the number of buffers stacked in the application thread's corresponding buffer queue is greater than 1, when the image content of the previous image (i.e., the sixth image) is completed, the application thread immediately renders the image content of the current image (i.e., the seventh image) without waiting for Vsync-app, and caches the rendered seventh image in the seventh buffer of the first queue. In other words, the rendering time of the current image is earlier than the time when Vsync-app is received. When the application thread performs frame interpolation, an N+1 timestamp can be used to render the Nth frame. When SF (Synchronous Frame) is pre-composited, the upper layer (i.e., the application thread) is notified to enable the frame interpolation scheme and interpolate the SF. This reduces the time when the number of buffers in the buffer queue is equal to 0 due to the pre-compositing of SF, thus improving image display performance.

[0194] Based on Figures 15 and 16 above, this application provides an image processing method, as shown in Figure 17, which is a flowchart of another SF pre-synthesis method provided by this application. Figure 17 is described mainly based on image frames, without distinguishing between images, synthesized images, and image frames.

[0195] S101. Count the number of buffers stacked in the SF's buffer queue.

[0196] Because SF has a cache queue for all layers, SF can sense how many buffers have accumulated for each application and are waiting to be consumed. That is, the image content in the buffer corresponding to the image frame needs to be composited. Based on this, the number of buffers stacked in SF's cache queue can be counted.

[0197] S102. Determine whether the application interface of the desktop application or the focused application is in a state of animation and whether the number of stacked buffers is greater than 1.

[0198] SF identifies motion events through the motion effect interface. By monitoring preset operation events (such as swiping the screen, clicking a button, touching the screen, folding the screen, etc.), it can determine whether the application interface has enabled continuous frame refresh operation, i.e., whether the application interface is in motion effect state, according to the preset rules of the operating system.

[0199] For example, as shown in Figure 1 above, when a desktop animation is detected, SF identifies the start and end of the animation through the desktop interface, thus determining that the desktop application is in an animation state. This pre-compositing scheme can be applied not only to animation scenarios but also to focused applications. For example, when a focused application is set, such as a chat application, and the user scrolls through a list in the chat application, detecting a scrolling operation, SF identifies through the application interface that the focused application's interface has started a continuous frame refresh operation. If it is determined that the number of buffers in the SF's cache queue for the chat application is greater than 1, the image processing scheme for requesting compositing provided in this application embodiment can be used to pre-composit the chat application's buffer to achieve image display. For example, when a focused application is set, such as a browser, to open an image in the browser, and the image zooms in and out, SF identifies through the application interface that the focused application's interface has started a continuous frame refresh operation. If it is determined that the number of buffers in the SF's cache queue for the browser is greater than 1, the image processing scheme for requesting compositing provided in this application embodiment can be used to pre-composit the browser's buffer to achieve image display.

[0200] If yes, then execute S103 to execute the pre-synthesis scheme. If no, then continue to execute S101 to count the number of buffers stacked in the SF's buffer queue and identify the animation.

[0201] It should be noted that when the number of buffers stacked in the SF's buffer queue is less than or equal to 1, if SF executes the pre-compositing scheme, the number of buffers may equal 0. If SF executes the pre-compositing scheme and there is any delay, a blank frame will be missed, resulting in frame drops. Therefore, the pre-compositing scheme should be enabled when the number of stacked buffers is greater than 1.

[0202] S103 and SF are immediately awakened, and the buffers in SF's buffer queue are synthesized in advance.

[0203] When SF detects that the application interface of the desktop application or the focused application is in a dynamic state and there are two or more buffers waiting to be consumed, SF is immediately woken up when the previous image frame of the current image frame is completed. Without waiting for the fence or the vsync signal, SF immediately performs compositing processing on the current image frame and pre-composites the buffers in SF's buffer queue.

[0204] S104. Move the fence of the current image frame and the next N image frames forward by one.

[0205] Under the condition that the application interface of the desktop application or the focused application is in a dynamic state and there are two or more buffers stacked, not only are the buffers in the SF cache queue synthesized in advance, but also the fence forwarding scheme is enabled. The fences under the native mechanism of the current image frame and the N image frames after the current image frame are all moved forward by one. That is, the offset value shift is incremented by 1 to obtain the updated offset value shift', so as to avoid the frame being unable to be passed to the hardware layer (i.e., the display driver) due to the underlying fence not being released.

[0206] S105. Send the Vsync signal to the application layer to pre-draw the image content of the next image frame.

[0207] Because SF (Frame Synthesis) compositing is a serial, blocking, and waiting process—meaning it's composited frame by frame—one implementation sends a Vsync signal (i.e., a frame insertion request) to the application thread in the application layer as soon as the pre-compositing scheme begins. Another implementation sends the Vsync signal to the application thread in the application layer only if the number of buffers in the SF's buffer queue is less than or equal to one.

[0208] For example, as shown in Figure 18, which is a schematic diagram of application frame interpolation provided in an embodiment of this application, the frame interpolation scheme of the application thread is illustrated through Figure 18. Since the number of buffers stacked in the SF buffer queue is greater than 1, an advance synthesis scheme can be executed, causing the number of buffers in the SF buffer queue to be consumed in a variation of 2, 1, 2, 1. Advance synthesis cannot be performed when the number of buffers is in a variation of 1, 0, 1, 0. If the SF executes the advance synthesis scheme, the number of buffers in the SF buffer queue will change to a variation of 1, 0, 1, 0. If there is any delay, a blank frame will occur, and then the number of buffers in the SF buffer queue will be consumed again in a variation of 2, 1, 2, 1, resulting in frame skipping. Therefore, the application thread also needs to perform frame interpolation so that the number of buffers in the SF buffer queue is always consumed in a variation of 2, 1, 2, 1. When the desktop application or the focus application's interface is in an animated state, and the number of buffers in the SF's buffer queue is consumed in a changing pattern of 1, 0, 1, 0, a frame insertion request is sent to the upper layer (i.e., the application thread). In other words, the condition for triggering the application thread to perform frame insertion is that the desktop application or the focus application's interface is in an animated state, and the number of buffers in the SF's buffer queue is less than 1. This is to prevent the upper layer from having only 1 buffer left in the SF's buffer queue due to the long rendering time of a certain frame, which would lead to frame loss during the display process.

[0209] For example, if the number of buffers in the SF's buffer queue is 3, and the startup effect is enabled, after pre-compositing one frame, the number of buffers in the SF's buffer queue is consumed in a pattern of 2, 1, 2, 1. In this case, if the SF requests frame interpolation from the upper layer, then after the upper layer frame interpolation, the number of buffers in the SF's buffer queue is consumed in a pattern of 3, 2, 3, 2. If the number of buffers in the SF's buffer queue is 2, and the startup effect is enabled, after pre-compositing one frame, the number of buffers in the SF's buffer queue is consumed in a pattern of 1, 0, 1, 0. In this case, if the SF requests frame interpolation from the upper layer, then after the upper layer frame interpolation, the number of buffers in the SF's buffer queue is consumed in a pattern of 2, 1, 2, 1.

[0210] Figure 18 illustrates the operation of the upper layer (i.e., the application thread) and SF when the SF proactive pre-compositing scheme is in effect. During normal rendering, the application thread receives the Vsync-app (shown as Vsync in Figure 18) and begins rendering the image content in the buffer. As shown in Figure 18, after rendering, the number of buffers in the SF's buffer queue increases by 1, and the SF takes the buffers for compositing processing, decreasing the number of buffers by 1. When the SF proactive pre-compositing scheme is enabled, it needs to send a frame interpolation request to the upper layer. The Vsync signal sent by the SF is different from the Vsync-app in the normal cycle; it is essentially a pseudo-Vsync-app, an additional Vsync-app. After receiving the Vsync signal sent by the SF, the application thread does not wait for the Vsync-app and renders the next frame of the current image frame. This process can be understood as a frame interpolation process. After frame interpolation is completed, the number of buffers in the SF's buffer queue increases by 1. By having the application thread pre-render the image frames, frame dropping caused by the SF consuming buffers in advance can be reduced.

[0211] In Figure 18 above, if SF determines whether the desktop application or the focused application's interface is in an animated state and the buffer count is equal to 2, after frame 1 is completed, it immediately forces frame 2 to be composited without waiting for the fence or Vsync. Because of the compositing process for frame 1, the buffer count decreases from 2 to 1. With the buffer count equal to 1, SF sends a Vsync signal to the application thread. The application thread, without waiting for Vsync-app, executes the frame interpolation process. After interpolation, the buffer count in SF's buffer queue increases by 1, making the buffer count equal to 2. SF then begins compositing frame 2, decreasing the buffer count from 2 to 1. If the application thread does not perform frame interpolation, then frame 1 is composited, decreasing the buffer count from 2 to 1, and then frame 2 is composited, decreasing the buffer count from 1 to 0.

[0212] S106. The animation ends, and the original fence mechanism is restored.

[0213] In this embodiment, SF identifies motion events through the motion effect interface (desktop interface or application interface). It can monitor preset operation events and determine whether the motion effect has ended based on operating system preset rules. When the motion effect ends, everything returns to its original state, that is, the native fence mechanism is restored, and the updated offset value shift' is adjusted to the offset value shift. At this point, the SF's pre-composition scheme in the motion effect is completed. Thus, after the motion effect is completed, the native fence mechanism is no longer interfered with, and the native fence mechanism is restored.

[0214] It should be noted that the updated offset value shift' can be configured to be effective only for a segment of continuous frame refresh. For example, during a segment of continuous frame refresh, if the application interface of the desktop or focused application is detected to be in animation mode (i.e., the application interface is performing continuous frame refresh), and the number of buffers in the SF's buffer queue is greater than 1, SF pre-composites the image content of the current image frame and increments the offset value shift of the current image frame and each subsequent image frame by 1 to obtain the updated offset value shift'. This ensures that the current image frame and each subsequent image frame are displayed based on the updated offset value shift', thus lengthening the SF's work-duration for the current image frame and each subsequent image frame. After this segment of continuous frame refresh ends, the next segment of continuous frame refresh still uses shift=static_cast.<size_t> ((mExpectedPresentTime-now) / VsyncPeriod) determines the shift value. If the number of buffers in the SF buffer queue is greater than 1, the shift value is then modified to update the shift value.

[0215] The image processing scheme for requesting synthesis provided in this application embodiment can solve the problem of frame dropping caused by long running time in Figure B of Figure 14. By sending a frame insertion request to the application thread of the application layer, the application thread performs frame interpolation (i.e., frame insertion). Because the work-duration is lengthened, the display time is delayed, as shown in Figure 19. Figure 19 is a schematic diagram of tracking another frame dropping problem provided in this application embodiment. The frame dropping problem shown in Figure A of Figure 19 corresponds to Figure B of Figure 14. In Figure A of Figure 19, the work-duration of SF is the time period from the start of synthesis of frame 1 to the completion of the display of frame 1. Because SF has a long running time in the process of synthesizing frame 3, when the display driver should be displaying frame 3, since frame 3 has not yet been synthesized, the display driver still displays the image frame before frame 3 (i.e., frame 2) during the display. Figure A of Figure 19 is represented by a dashed box, which shows the frame dropping phenomenon and the display stuttering. In Figure B of Figure 19, due to the pre-compositing scheme, the image content of the image frames preceding frame 1 is pre-composited by SF. Since the pre-composited frames occupy the display space, the image frames preceding frame 1 (frame 0) are displayed. Then, frame 1 and subsequent frames 2, 3, 4, etc., need to be displayed sequentially, which lengthens the SF's work-duration. The display position corresponding to frame 0 corresponds to frame 1 without the pre-compositing scheme. Therefore, even if SF experiences a long runtime during the compositing of frame 3, the display driver can still display frame 3 normally, thus reducing frame drops. In other words, when executing the pre-compositing scheme, the image content of the image frames is pre-composited by SF, and even a long running time will not cause frame drops.

[0216] Electronic device business systems can pre-embed events such as potential business failures and user behaviors based on their own business logic. This allows them to detect business failures or user behaviors during process execution and mark them (also known as event tracking). For example, in Figure 9 above, if the buffer cannot be transferred during the display delivery process, the application thread may not be able to obtain the buffer, increasing the rendering time between two frames. In this case, the business system will mark the event. Similarly, in Figure 13 above, if the display driver layer detects that the display delivery is delayed and the delivery time is longer, the business system will mark the event. By executing a pre-compositing scheme, the buffer is easier to transfer. In Figure 9, the application thread can obtain the released buffer in time and will not be detected, so no event tracking is needed, or detection and tracking are performed but not reported, thus reducing the power consumption of the electronic device. In the display driver layer of Figure 13, although the display delivery time is longer, no detection and tracking are needed, or detection and tracking are performed but not reported, thus reducing the power consumption of the electronic device.

[0217] In this embodiment, when the application interface of the desktop application or the focused application is identified as being in motion effect mode and the number of buffers in the SF's cache queue is greater than 1, the SF pre-composites the image content of the image frame. This is equivalent to ignoring the native fence mechanism, pre-compositing, and shifting the fence of the current frame and all subsequent frames forward by one position to avoid the frame being unable to be passed to the hardware layer (i.e., the display driver) due to the underlying fence not being released. At the same time, when the pre-compositing scheme is executed, a frame insertion request is also sent to the upper layer (i.e., the application thread) to drive the upper layer to pre-draw and render the image frame. This provides more work margin for the composite and display of the motion effect, improves the fault tolerance in the display process, reduces frame dropping, and ensures smooth screen display, thereby improving image display performance.

[0218] It should be noted that when SF detects that the application interface of the desktop application or the focused application is in an animated state and the number of buffers is greater than 1, it immediately performs compositing processing on the current image frame while completing the compositing task of the previous image frame. The number of buffers is reduced by 1, which is equivalent to inserting a frame during the compositing process. However, this frame insertion does not change the frame rate. Unlike frame insertion in game design, the purpose of this application embodiment is to consume the stacked buffers in advance, prevent buffer stacking, lengthen the work-duration of each subsequent image frame, improve fault tolerance, and reduce the occurrence of frame drops.

[0219] The image processing scheme for requesting compositing provided in this application is similar to a frame stabilization scheme. When buffers are stacked in the buffer queue of the SF, the SF needs to consume the stacked buffers in advance and send them to the display screen for display. If the SF trace in a certain scheme (such as the trace shown in Figure 15 above) indicates that when buffers are stacked in the SF's buffer queue, the composited image frame is sent to the buffer queue corresponding to the display driver in advance and waits on the display driver side without affecting the normal operation of the SF, then it indicates that the scheme adopts the image processing scheme for requesting compositing provided in this application.

[0220] Based on Figures 15-19 above, this application provides an image processing method, as shown in Figure 20.

[0221] S201 and SF determine the number of buffers stacked in the first queue when the electronic device is in a preset scenario.

[0222] The preset scenarios include target application launch scenario, target application exit scenario, switching between multiple pages in the desktop application of an electronic device, switching between multiple applications via the desktop application, folding scenario when the electronic device is a foldable device, unfolding scenario when the electronic device is a foldable device, continuous sliding scenario of the application interface of the focus application, and scenario where the interface elements in the application interface of the focus application change, etc.

[0223] The first queue indicates the cache queue of SF. The first queue is the cache queue corresponding to the image to be composited for the desktop application or the focus application. The stacked buffers in the first queue are used to cache the image to be composited after drawing and rendering.

[0224] S202. When the number of buffers stacked in the first queue is greater than 1, within the first Vsync cycle, SF performs compositing processing on the first image obtained from the first buffer in the first queue, and after completing the compositing processing on the first image, it begins to perform compositing processing on the second image obtained from the second buffer in the first queue.

[0225] The first Vsync cycle is the Vsync cycle used to wake up the SF to perform compositing processing on the first buffer. The start time of the first Vsync cycle is the moment when the SF receives the first Vsync message. Upon receiving the first Vsync message, the SF retrieves the first image from the first buffer in the first queue and performs compositing processing on the first image to obtain the first composite image.

[0226] When the application interface of the desktop application or the focused application displays animation effects and the number of buffers stacked in the first queue is greater than 1, after completing the compositing process of the first image, the second image is obtained from the second buffer in the first queue in the first Vsync cycle, and the compositing process of the second image is started to obtain the second composite image.

[0227] The animation displayed on the desktop or featured application interface indicates that the application interface is continuously refreshing frames, requiring the pre-compositing of image frames to reduce frame drops. In the animation scene, if the number of buffers stacked in the first queue is greater than one, after the first image in the first buffer is composited, the second image in the second buffer is immediately composited without waiting for the second Vsync message or the fence. This is equivalent to forcibly pre-compositing the second buffer to obtain the second composite image. The second buffer is the next buffer adjacent to the first buffer. This extends the working time of SF, increases the fault tolerance of image frame composite delivery, reduces frame drops, and improves image display performance. Moreover, because the working time of SF is extended, events such as delayed delivery or buffer overload will not be recognized, thus eliminating the need for tracking or reporting, and reducing the power consumption of electronic devices.

[0228] In some embodiments, S202 can be implemented as follows: When the application interface of the desktop application or the focused application displays animation effects, and the number of buffers stacked in the first queue is greater than 1, after the first image in the first buffer is composited, the second image is obtained from the second buffer in the first queue in the first Vsync cycle. The second image is composited in the first Vsync cycle or the second Vsync cycle to obtain the second composited image. The second Vsync cycle is the next Vsync cycle after the first Vsync cycle. The start time of compositing the second buffer is earlier than the time when the SF receives the second Vsync message. The second Vsync message is a Vsync message generated after the first Vsync message. Thus, before the second Vsync message is received, the SF has already started compositing the second image in the second buffer, realizing an early compositing scheme. Compared to the native compositing process that requires waiting for Vsync to wake up and for the fence to be released, this extends the working time of the SF, improves the fault tolerance of the display, and thus reduces the probability of frame drops.

[0229] In some embodiments, after the second image in the second buffer is composited, while the application interface of the desktop application or the focused application is in motion effect state, the system waits for the next Vsync message (i.e., the second Vsync message) to wake it up. During the second Vsync cycle, the system retrieves the third image from the third buffer in the first queue and performs composite processing on the third image to obtain the third composite image. Then, the system waits for the next Vsync message (i.e., the third Vsync message) to wake it up. During the third Vsync cycle, the system performs composite processing on the fourth image retrieved from the fourth buffer in the first queue to obtain the fourth composite image, until the application interface of the desktop application or the focused application completes the motion effect.

[0230] By enabling the pre-compositing scheme, SF pre-composites the image content in the second buffer and subsequent buffers. Even if the number of buffers stacked in the first queue is no more than one, the pre-compositing scheme remains in effect. Regardless of the number of buffers stacked in the first queue, the compositing process for each buffer is as follows: SF waits for the next Vsync signal to wake it up, and without waiting for the fence, performs compositing processing on the image content in the buffer. This allows for the early consumption of stacked buffers, preventing buffer stacking, and lengthens the time from compositing to display for each subsequent image frame, resulting in higher fault tolerance and reduced frame drops.

[0231] S203. When the number of buffers stacked in the first queue is 1, within the first Vsync cycle, SF performs synthesis processing on the first image obtained from the first buffer in the first queue to obtain the first synthesized image.

[0232] When the number of buffers stacked in the first queue is 1, SF cannot execute the pre-compositing scheme. In the first Vsync cycle, the first image obtained from the first buffer in the first queue is composited to obtain the first composite image.

[0233] After S203, S201 continues to be executed, that is, SF continues to detect the number of buffers stacked in the first queue. If the number of buffers stacked in the first queue is greater than 1, the proposed synthesis scheme is executed to reduce the occurrence of frame dropping.

[0234] S204. If the number of buffers stacked in the first queue is greater than 1, SF updates the offset value to obtain the updated offset value.

[0235] It should be noted that the execution order of S202 and S204 is not important, and they can be executed simultaneously. For example, if the application interface of the desktop application or the focused application displays animation effects, and the number of buffers stacked in the first queue is greater than 1, the compositing process of the second image in the second buffer will begin immediately after the first image in the first buffer is composited, and the offset value will also be updated at the same time.

[0236] Under the native fence mechanism, the frame forwarding (SF) determines which frame to wait for based on the offset value. If the fence of the image frame that the frame is waiting for is released when the Vsync signal is received, the image content in the buffer corresponding to that frame is composited. However, in this embodiment, an early composite scheme is enabled. The SF does not wait for the Vsync signal or the fence, forcibly composites one frame in advance. Then, the SF waits for the Vsync signal but not for the fence, and composites the image content in the subsequent buffer. After the composite is completed, it still has to wait for the fence to be released before it can be sent to the display driver for display. Based on this, the offset value is updated so that the second composite image and the N composite images after the second composite image are all sent for display according to the updated offset value. Due to the early composite, the display time of the second composite image and the N composite images after it is extended during the display process, the fault tolerance of the display process is higher, the frame dropping phenomenon is reduced, the screen is not stuttered, and thus the image display performance is improved.

[0237] S205, SF determines the first target image frame based on the updated offset value and the second composite image corresponding to the second image; the first target image frame is displayed before the second target image frame determined based on the second composite image and the offset value.

[0238] S206. The display driver sends the second composite image to the display when the fence of the first target image frame is released.

[0239] For the second composite image, the first target image frame is determined based on the updated offset value. When the updated offset value is 1, the first target image frame is the two frames preceding the second composite image (i.e., the frame preceding the frame preceding the frame preceding the second composite image). With the fence of the two frames preceding the second composite image released, the display driver sends the second composite image for display. When the updated offset value is 2, the first target image frame is the three frames preceding the second composite image (i.e., the frame preceding the two frames preceding the second composite image). With the fence of the three frames preceding the second composite image released, the display driver sends the second composite image for display. This process of sending the second composite image for display has higher fault tolerance, reduces frame drops, and prevents screen stuttering, thereby improving image display performance.

[0240] Compared to related technologies where the display fence for the current image frame is not released, and SF (Synchronous Rendering) needs to wait for the next Vsync before attempting to composite the image content of the current image frame, this application's embodiment, by pre-processing the image content of the current image frame, not only extends the working time of SF, but also improves the fault tolerance of image frame synthesis and display, reducing frame drop. Furthermore, because the working time of SF is extended, events such as delayed display or buffer overflows will not be recognized, thus eliminating the need for reporting or intervention. It can also slow down the system scheduling frequency; for example, the operating frequency of the processing chip (CPU or GPU, etc.) can be appropriately reduced, thereby lowering the power consumption of the electronic device.

[0241] Based on Figure 7 and Figures 15-20 above, embodiments of this application provide a method for interaction between threads in an electronic device.

[0242] For example, as shown in Figure 21, which is an interaction flowchart between threads (application thread, SF, HWC, and display driver) in an electronic device according to an embodiment of this application, the application thread can be a desktop application or a focused application. The application thread is located in the application layer of Figure 7, the SF is located in the Framework layer of Figure 7, the hardware compositor (HWC) is located in the HAL layer of Figure 7, and the display driver is located in the kernel layer of Figure 7. The application thread's buffer queue includes buffer11, buffer12, buffer13, buffer14, buffer15, etc.; the SF's buffer queue (i.e., the first queue) includes buffer21, buffer22, buffer23, buffer24, buffer25, etc. The application thread and SF's buffer queues are numbered according to their respective buffer order. After the application thread renders the buffer, it sends it to the SF's buffer queue. The application thread and SF may have the same buffer.

[0243] In Figure 21, Vsync signal 1 corresponds to period 1, Vsync signal 2 to period 2, Vsync signal 3 to period 3, and Vsync signal 4 to period 4. The Vsync signals include Vsync-app (for the application thread), Vsync-sf (for the SF), and Vsync-hw (for the display driver). All three have the same period, but their start times may differ slightly, as shown in Figures 9-16 and 18-19 above. There is an offset between the start times of Vsync-sf and Vsync-hw; similarly, as shown in Figures 9 and 18 above, there is an offset between the start times of Vsync-app and Vsync-sf. Figure 8 illustrates the process of image frame rendering and display from drawing to display using the alignment of the start times of Vsync-app, Vsync-sf, and Vsync-hw as an example, but it does not represent the actual start times. Here, Figure 21 uses the alignment of the start times of Vsync-app, Vsync-sf, and Vsync-hw as an example for illustration, without distinguishing their actual start times.

[0244] S11. The application thread is awakened by Vsync-app1 and begins to draw rendering buffer11.

[0245] S12, The application thread sends the rendered buffer11 to the cache queue of SF.

[0246] S13 and SF are woken up by Vsync-sf1, detect that the fence has been released, and send buffer21 to HWC.

[0247] SF is woken up by Vsync-sf1, detects that the fence of the image frame corresponding to buffer21 calculated based on the shift value has been released, retrieves the image content in buffer21 from SF's buffer queue and sends it to HWC.

[0248] S14, HWC begins synthesizing buffer21.

[0249] S15, HWC sends display buffer21.

[0250] During the native compositing process, in cycle 1, SF, awakened by Vsync-sf1, reads the image content from buffer21 and sends the image content to HWC. HWC then performs compositing processing on the image content in buffer21 after rendering and transmits it to the display driver for display.

[0251] S16. The display driver sequentially displays the image frames corresponding to the buffer.

[0252] The above S11-S16 illustrate the image frame display process under period 1 and the native fence mechanism. SF needs to wait for Vsync-sf to wake up and for the fence to be released before performing compositing. The application thread sends the rendered buffer11 to the SF's buffer queue for queuing; SF sends the composited buffer21 to the display driver's buffer queue for queuing, and the display driver sequentially displays the buffers in its buffer queue.

[0253] It should be noted that the above-mentioned application thread's drawing and rendering steps S11-S12, SF's compositing steps S13-S15, and display driver's display step S16 can be executed in parallel. Figure 21 is only an illustrative illustration and does not represent the actual execution order.

[0254] S17. The application thread is awakened by Vsync-app2 and begins to draw rendering buffer12.

[0255] S18, The application thread sends the rendered buffer12 to the cache queue of SF.

[0256] S19 and SF are woken up by Vsync-sf2, detect that the fence has been released, and send buffer22 to HWC.

[0257] S20 and HWC begin synthesizing buffer22.

[0258] S21, HWC sends display buffer22.

[0259] S22, SF determines that the application interface of the desktop application or the focused application has an animation effect, and the number of buffers stacked in the cache queue of SF is greater than 1.

[0260] S23 and SF immediately send buffer23 to HWC.

[0261] S24 and HWC begin synthesizing buffer23.

[0262] In cycle 2, if the desktop application or the focused application's interface displays animations (e.g., launch, exit, switch, or swipe), and the number of buffers stacked in the SF's cache queue is greater than 1, it indicates that the pre-compositing conditions are met, and the pre-compositing scheme begins execution. After the image content in buffer22 is composited, SF does not wait for Vsync-sf3 to wake up, nor for the fence of the image frame corresponding to buffer23 to be released. It immediately reads the image content from buffer23 and sends the image content to HWC. HWC then performs compositing processing on the image content in buffer23 that has been rendered and passes it to the display driver for display.

[0263] In cycle 2, HWC begins compositing the image content in buffer23, and the compositing can be completed in cycle 2. The compositing of buffer23 begins before Vsync-sf3.

[0264] S25, HWC sends display buffer23.

[0265] When the pre-compositing conditions are met, SF also updates the offset value shift to obtain the updated shift value, and calculates the image frame corresponding to buffer23 based on the updated shift value. When SF detects that the fence of the image frame corresponding to buffer23 has been released, HWC passes the image frame corresponding to buffer23 to the display driver for display.

[0266] S26. The display driver sequentially displays the image frames corresponding to the buffer.

[0267] When the conditions for early synthesis are met, SF not only immediately retrieves the image content of buffer23 from SF's buffer queue and sends it to HWC, but also updates the offset value. These two processes can be performed simultaneously, or the offset value can be updated first, followed by reading the image content of buffer23, or vice versa. This application does not impose any restrictions on this.

[0268] In this embodiment, when the pre-compositing scheme is met, SF does not need to wait for Vsync-sf to wake up or for fence to be released. Combined with HWC, the image content in buffer23 is immediately composited, and the shift value is updated. This ensures that the image frame corresponding to buffer23 is delivered to the display driver for display when the fence corresponding to the image frame in buffer23, calculated based on the updated shift value, is released. Compared to the native compositing process, which requires waiting for Vsync-sf to wake up and for fence to be released, this extends the working time of SF, resulting in higher fault tolerance for image frame compositing and display, reduced frame dropping, and improved image display performance. Furthermore, because the time from compositing to display is extended, events such as delayed display or buffer overflows are not recognized, eliminating the need for reporting or notifying, thus reducing the power consumption of the electronic device.

[0269] The above S17-S26 illustrate the display flow of the first cycle of the image frame under the pre-compositing scheme with cycle 2. SF does not need to wait for Vsync-sf to wake up or for the fence to be released; it immediately performs compositing processing. The application thread sends the rendered buffer12 to the SF's buffer queue for queuing; the SF sends the composited buffer22 to the display driver's buffer queue for queuing, and immediately after compositing buffer22, it composites buffer23 and sends the composited buffer23 to the display driver's buffer queue for queuing; the display driver sequentially displays the buffers in its buffer queue. In cycle 2, the application thread renders once, the SF composites twice, and the display driver displays once.

[0270] It should be noted that the drawing and rendering steps S17-S18 of the application thread, the compositing steps S19-S25 of SF, and the display step S26 of the display driver can be executed in parallel. Figure 21 is only an illustrative illustration and does not represent the actual execution order.

[0271] S27. The application thread is awakened by Vsync-app3 and begins to draw rendering buffer13.

[0272] S28. The application thread sends the rendered buffer13 to the cache queue of SF.

[0273] S29, SF is woken up by Vsync-sf3 and sends buffer24 to HWC.

[0274] S30 and HWC begin synthesizing buffer24.

[0275] S31, HWC sends display buffer24.

[0276] When the fence of the image frame corresponding to buffer24, calculated based on the updated shift value, is released, HWC passes the image frame corresponding to buffer24 to the display driver for display.

[0277] S32, the display driver sequentially displays the image frames corresponding to the buffer.

[0278] In this embodiment, after the SF pre-synthesizes the image content in buffer 23, regardless of the number of buffers stacked in the SF's buffer queue, the display steps S27-S31 described above are continued. That is, without waiting for the fence and waiting for the next Vsync-sf (e.g., Vsync-sf3) to wake up, the image content in the next buffer (e.g., buffer 24) is started to be synthesized. Compared with the native synthesis process that requires waiting for the Vsync-sf to wake up and for the fence to be released, the working time of the SF is still extended, the fault tolerance of image frame synthesis and display is higher, frame dropping is reduced, and image display performance is improved.

[0279] The above steps S27-S32 illustrate the display flow of the second cycle of the image frame under the pre-compositing scheme in cycle 3. SF needs to wait for Vsync-sf to wake up for compositing processing. The application thread sends the rendered buffer13 to the SF's buffer queue for queuing; the SF sends the composited buffer24 to the display driver's buffer queue for queuing; the display driver sequentially displays the buffers in its buffer queue. Specifically, in cycle 3, the application thread renders once, the SF composites once, and the display driver displays once. Subsequent cycles follow the same display flow as cycle 3 and will not be described further here.

[0280] It should be noted that the above-mentioned application thread's drawing and rendering steps S27-S28, SF's compositing steps S29-S31, and display driver's display step S32 can be executed in parallel. Figure 21 is only an illustrative illustration and does not represent the actual execution order.

[0281] S33. The application thread is awakened by Vsync-app3 and begins to draw rendering buffer14.

[0282] S34. The application thread sends the rendered buffer14 to the SF's cache queue.

[0283] During the native drawing and rendering process, the application thread, awakened by Vsync-app1, begins to draw and render the image content in buffer14, and caches the rendered buffer14 in SF's cache queue.

[0284] S35, SF determines that the number of buffers stacked in SF's buffer queue is less than 1.

[0285] S36 and SF send frame insertion requests to the application thread.

[0286] The application thread is above SF. In the pre-compositing scheme, the number of buffers in the SF's buffer queue may change from 0 to 1 to 0 to 1. If there is any delay, a blank frame will appear and not be displayed, resulting in frame skipping. Therefore, as the application thread above SF, it also needs to perform frame interpolation. SF sends a frame interpolation request to the application thread to start the frame interpolation scheme.

[0287] S37, The application thread immediately draws the rendering buffer 15.

[0288] Based on the frame interpolation request, the application thread immediately begins to draw and render the image content in buffer15 after completing the drawing and rendering of the image content in buffer14.

[0289] S38. The application thread sends the rendered buffer15 to the SF's cache queue.

[0290] S39 and SF are woken up by Vsync-sf3 and send buffer25 to HWC.

[0291] S40 and HWC begin synthesizing buffer25.

[0292] S41, HWC sends display buffer 25.

[0293] S42, The display driver sequentially displays the image frames corresponding to the buffer.

[0294] S33-S42 above illustrates the display flow of an image frame in a certain period under the pre-compositing scheme of period 4. The SF sends a frame interpolation request to the application thread, causing the application thread to pre-draw the rendering buffer. The application thread sends the rendered buffer 14 to the SF's buffer queue for queuing. Upon receiving the frame interpolation request, after drawing rendering buffer 14, it immediately draws rendering buffer 15 and sends the rendered buffer 15 to the SF's buffer queue for queuing. The SF sends the composited buffer 25 to the display driver's buffer queue for queuing. The display driver then sequentially displays the buffers in its buffer queue. In period 4, the application thread renders twice, the SF composites once, and the display driver displays once. The subsequent application thread's response to the frame interpolation request follows the same display flow as period 4 and will not be described further here.

[0295] It should be noted that the drawing and rendering steps S33-S38 of the application thread, the compositing steps S39-S41 of SF, and the display step S42 of the display driver can be executed in parallel. Figure 21 is only an illustrative illustration and does not represent the actual execution order.

[0296] The application thread begins frame interpolation. After the image content in buffer14 is composited, the application thread does not wait for Vsync-app5 to wake it up. It immediately begins to draw and render the image content in buffer15 and caches buffer15 in SF's cache queue.

[0297] The application thread begins drawing and rendering the image content in buffer15 at the start of cycle 4, and can complete the drawing and rendering process at the start of cycle 4. The drawing and rendering of buffer15 occurs before Vsync-app5.

[0298] The application thread pre-draws and renders the image content in the buffer in the application thread's cache queue, so as to pass the completed drawing and rendering buffer to the SF cache queue in advance. This reduces the frame drop phenomenon caused by only one buffer remaining in the SF cache queue, thus improving image display performance.

[0299] It should be noted that the rendering process in the application thread, the compositing process in SF, and the display process in the display driver are executed simultaneously. Figure 21 above shows the execution processes of the application thread, SF, and display driver respectively, but does not represent the actual execution order. That is, at a certain time, the electronic device executes not only the rendering process, but also the compositing process and the display process. S33-S42 can be at any stage in the pre-compositing process. The frame interpolation process of the application thread is executed only when an interpolation request is initiated by SF. There is no limit to the number of times the application thread performs frame interpolation, that is, S33-S42 may not be executed, or it may be executed once or multiple times. As long as the condition that the number of buffers stacked in the SF's buffer queue is less than 1 is met, SF sends an interpolation request to the application thread to drive the upper layer to perform pre-rendering of image frames. This can leave more work margin for the compositing and display in the animation, improve the fault tolerance in the display process, reduce frame dropping, and ensure smooth screen display, thereby improving image display performance.

[0300] It is understood that the number of buffers stacked in the SF's buffer queue is not less than 1, and the SF can also send frame interpolation requests to the application thread; this embodiment of the application does not impose any restrictions on this. For example, after the SF synthesizes a frame in advance, the number of buffers in the SF's buffer queue is consumed in a pattern of 2, 1, 2, 1. In this case, if the SF requests frame interpolation from the upper layer, then after the upper layer frame interpolation, the number of buffers in the SF's buffer queue is consumed in a pattern of 3, 2, 3, 2. Alternatively, after the SF synthesizes a frame in advance, the number of buffers in the SF's buffer queue is consumed in a pattern of 1, 0, 1, 0. In this case, if the SF requests frame interpolation from the upper layer, then after the upper layer frame interpolation, the number of buffers in the SF's buffer queue is consumed in a pattern of 2, 1, 2, 1.

[0301] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0302] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0303] This application also provides a chip system including one or more processors; when the chip system is running, the one or more processors can execute computer instructions to cause the chip system to perform the image processing methods in the above-described method embodiments.

[0304] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0305] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0306] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0307] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. An image processing method, the method being applied to an electronic device, characterized in that, include: Receive the user's first action; In response to the first operation, start or exit the target application; The method includes the following steps during the process of starting or exiting the target application: Determine the number of buffers stacked in the first queue; If the number of buffers stacked in the first queue is greater than 1, during the first Vsync cycle, the image synthesizer performs synthesis processing on the first image obtained from the first buffer in the first queue to obtain the first synthesized image, and after completing the synthesis processing on the first image, it begins to perform synthesis processing on the second image obtained from the second buffer in the first queue.

2. The method of claim 1, wherein, The method further includes: The image synthesizer identifies whether the target application is started or exited based on the event points provided by the business system.

3. The method of claim 1 or 2, wherein, The second image obtained from the second buffer in the first queue is subjected to compositing processing, including: The image synthesizer acquires the second image from the second buffer during the first Vsync cycle, and completes the synthesis processing of the second image during the first Vsync cycle or the second Vsync cycle to obtain the second synthesized image; The second Vsync cycle is located after the first Vsync cycle. The start time of the first Vsync cycle is the time when the image synthesizer receives the first Vsync message. The start time of the second image synthesis is before the time when the image synthesizer receives the second Vsync message. The second Vsync message is a Vsync message generated after the first Vsync message.

4. The method of claim 3, wherein, The method further includes, during the process of starting or exiting the target application: During the second Vsync cycle, the image synthesizer performs synthesis processing on the third image obtained from the third buffer in the first queue to obtain the third synthesized image; During the third Vsync cycle, the image synthesizer performs synthesis processing on the fourth image obtained from the fourth buffer in the first queue to obtain the fourth synthesized image.

5. The method according to any one of claims 1 to 4, wherein After the target application is launched or exited, the method further includes: If the fence corresponding to the fifth image is detected to be released, the image synthesizer performs synthesis processing on the fifth image obtained from the fifth buffer of the first queue within the fourth Vsync cycle to obtain the fifth synthesized image.

6. The method according to any one of claims 1 to 5, wherein, After determining the number of buffers stacked in the first queue, the method further includes: When the number of buffers stacked in the first queue is 1, the image synthesizer performs synthesis processing on the first image obtained from the first buffer in the first queue during the first Vsync cycle to obtain the first synthesized image.

7. The method according to any one of claims 1 to 6, wherein If the number of buffers stacked in the first queue is greater than 1, the method further includes: The image synthesizer updates the offset value to obtain an updated offset value; A first target image frame is determined based on the updated offset value and the second composite image corresponding to the second image; the first target image frame is displayed before the second target image frame determined based on the second composite image and the offset value; The display driver sends the second synthesized image to the display when the fence of the first target image frame is released.

8. The method as described in claim 7, characterized in that, After displaying the second synthesized image, the method further includes: The image synthesizer determines the target image frame for each of the N synthesized images based on the updated offset value and the N synthesized images following the second synthesized image. The display driver sequentially displays the N composite images when the fence of the target image frame of each of the N composite images is released.

9. The method as described in claim 7 or 8, characterized in that, The image synthesizer updates the offset value to obtain an updated offset value, including: The image synthesizer increments the offset value by one to obtain the updated offset value.

10. The method as described in claim 7 or 8, characterized in that, The display driver displays the second synthesized image when the fence of the first target image frame is released, including: If the update offset value is 1, then the display driver will send the second composite image to be displayed when the fence of the first two frames of the second composite image is released. The first two frames refer to the frame before the previous frame of the second composite image. If the update offset value is 2, then the display driver will send the second composite image to be displayed when the fence of the first three frames of the second composite image is released, where the first three frames refer to the frame before the first two frames.

11. The method according to any one of claims 7-10, characterized in that, The image synthesizer updates the offset value, and after obtaining the updated offset value, the method further includes: When the image synthesizer recognizes that the target application has been launched or exited based on the event points provided by the business system, it updates the offset value to the specified offset value.

12. The method according to any one of claims 1-11, characterized in that, The method further includes, during the process of starting or exiting the target application: The desktop application performs drawing and rendering processing on the image content and caches the completed image content in the buffer of the first queue; the first queue is the cache queue corresponding to the image to be synthesized by the desktop application; After performing composite processing on the second image obtained from the second buffer in the first queue, the method further includes: If the number of buffers stacked in the first queue is less than 1, the image synthesizer sends a frame interpolation request to the desktop application; In response to the frame interpolation request, the desktop application caches the sixth image obtained from rendering in the first queue in the sixth buffer within the current Vsync cycle, and then starts the next rendering in the current Vsync cycle, caching the seventh image obtained from rendering in the first queue in the seventh buffer.

13. An image processing method, said method being applied to an electronic device, characterized in that, The method includes: Receive the user's second action; In response to the second operation, switching is performed between multiple pages in the desktop application, or switching is performed between multiple applications via the desktop application; In the process of switching between multiple pages or multiple applications, the method includes: Determine the number of buffers stacked in the first queue; If the number of buffers stacked in the first queue is greater than 1, during the first Vsync cycle, the image synthesizer performs synthesis processing on the first image obtained from the first buffer in the first queue to obtain the first synthesized image, and after completing the synthesis processing on the first image, it starts to perform synthesis processing on the second image obtained from the second buffer in the first queue.

14. The method as described in claim 13, characterized in that, During the switching process between multiple pages, or between multiple applications, the method further includes: The desktop application performs drawing and rendering processing on the image content and caches the completed image content in the buffer of the first queue; the first queue is the cache queue corresponding to the image to be synthesized by the desktop application; After performing composite processing on the second image obtained from the second buffer in the first queue, the method further includes: If the number of buffers stacked in the first queue is less than 1, the image synthesizer sends a frame interpolation request to the desktop application; In response to the frame interpolation request, the desktop application caches the sixth image obtained from rendering in the first queue in the sixth buffer within the current Vsync cycle, and then starts the next rendering in the current Vsync cycle, caching the seventh image obtained from rendering in the first queue in the seventh buffer.

15. An image processing method, the method being applied to an electronic device, characterized in that, The electronic device is a foldable device, and the method includes: Receive the user's third action; In response to the third operation, the foldable device is folded or unfolded; The method includes the following steps during the folding or unfolding of the foldable device: Determine the number of buffers stacked in the first queue; If the number of buffers stacked in the first queue is greater than 1, during the first Vsync cycle, the image synthesizer performs synthesis processing on the first image obtained from the first buffer in the first queue to obtain the first synthesized image, and after completing the synthesis processing on the first image, it starts to perform synthesis processing on the second image obtained from the second buffer in the first queue.

16. The method as described in claim 15, characterized in that, In response to the third operation, folding or unfolding the foldable device includes: In response to the third operation, the display interface of the foldable device is switched from an unfolded state to a folded state, or... In response to the third operation, the display interface of the foldable device is switched from the folded state to the unfolded state; The display interface is either the interface corresponding to the desktop application or the interface corresponding to the focused application.

17. The method as described in claim 15 or 16, characterized in that, During the process of folding or unfolding the foldable device, the method further includes: The desktop application or the focused application performs drawing and rendering processing on the image content and caches the drawn and rendered image content in the buffer of the first queue; the first queue is the cache queue corresponding to the image to be synthesized by the desktop application or the focused application; After performing composite processing on the second image obtained from the second buffer in the first queue, the method further includes: If the number of buffers stacked in the first queue is less than 1, the image synthesizer sends an interpolation request to the desktop application or the focused application. In response to the frame interpolation request, the desktop application or the focused application caches the sixth image obtained from drawing and rendering in the sixth buffer of the first queue within the current Vsync cycle, and then starts the next drawing and rendering within the current Vsync cycle, and caches the seventh image obtained from drawing and rendering in the seventh buffer of the first queue.

18. An image processing method, the method being applied to an electronic device, characterized in that, The method includes: Receive the user's fourth action; In response to the fourth operation, the application interface of the focused application is continuously swiped, or the interface elements in the application interface of the focused application change. The method includes the following steps during the continuous swiping of the application interface of the focused application, or during the change of interface elements in the application interface of the focused application: Determine the number of buffers stacked in the first queue; If the number of buffers stacked in the first queue is greater than 1, during the first Vsync cycle, the image synthesizer performs synthesis processing on the first image obtained from the first buffer in the first queue to obtain the first synthesized image, and after completing the synthesis processing on the first image, it starts to perform synthesis processing on the second image obtained from the second buffer in the first queue.

19. The method as described in claim 18, characterized in that, During the continuous swiping of the application interface of the focused application, or during the process of changes in the interface elements of the application interface of the focused application, the method further includes: The focus application performs drawing and rendering processing on the image content and caches the completed image content in the buffer of the first queue; the first queue is the cache queue corresponding to the image to be synthesized by the focus application; After performing composite processing on the second image obtained from the second buffer in the first queue, the method further includes: If the number of buffers stacked in the first queue is less than 1, the image synthesizer sends an interpolation request to the focus application; In response to the frame interpolation request, the focus application caches the sixth image obtained from drawing and rendering in the sixth buffer of the first queue within the current Vsync cycle, and then starts the next drawing and rendering within the current Vsync cycle, caching the seventh image obtained from drawing and rendering in the seventh buffer of the first queue.

20. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1-19.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-19.

22. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1-19.

23. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-19.

Images