Image processing method, and electronic device

Abstract

An image processing method, and an electronic device, which relate to the technical field of terminals. The method comprises: at a first time point, starting compositing of a first image; at a second time point, starting compositing of a second image, wherein the second time point is after the first time point; and at a third time point, starting compositing of a third image, wherein the third time point is after the second time point, the first image, the second image and the third image are three consecutive image frames, the second image is the next image frame of the first image, and the third image is the next image frame of the second image. From the first time point to the third time point, a frame rate remains unchanged, and a first time interval between the first time point and the second time point is different from a second time interval between the second time point and the third time point. In this way, the start time for compositing of each image frame can be flexibly adjusted, such that both low display latency and display smoothness are taken into consideration, thereby improving display performance.

Description

An image processing method and electronic device

[0001] This application claims priority to Chinese Patent Application No. 202510300270.2, filed on March 13, 2025, entitled "An Image Processing Method and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Technology

[0003] Image processing typically includes the following steps: rendering, compositing, and display. Among these, the electronic device can simulate the vertical synchronization (VSYNC) signal (often denoted as VSYNC-SF) of the SurfaceFlinger based on the display time point. Subsequently, the electronic device can begin compositing when the VSYNC-SF arrives, ensuring that a composited image is available for display when the display time point arrives.

[0004] However, using the above image processing scheme may result in high display latency or display stuttering. Display latency refers to the time from the start of drawing and rendering an image frame to the time the image is displayed. High display latency may lead to high response delay, such as a long delay in displaying handwriting in a stylus scenario. Summary of the Invention

[0005] This application provides an image processing method and an electronic device that can dynamically adjust VSYNC-SF, thereby balancing low display latency and smooth display, and improving display performance.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, an image processing method is provided, applied to an electronic device. Specifically, the method includes: at a first moment, starting to synthesize a first image; at a second moment, starting to synthesize a second image, the second moment being after the first moment; at a third moment, starting to synthesize a third image, the third moment being after the second moment, wherein the first image, the second image, and the third image are three consecutive frames, the second image being the next frame after the first image, and the third image being the next frame after the second image. The frame rate remains constant from the first moment to the third moment, and the first time interval between the first and second moments is different from the second time interval between the second and third moments.

[0008] For example, in Figure 11 or Figure 12 below, the first image can be image I, the second image is image j, and the third image is image K. Correspondingly, the first time step is SF1, the second time step is SF2j, and the third time step is SF3k.

[0009] In summary, by employing this application, even if the frame rate remains unchanged, the time interval between the start of image synthesis for two adjacent frames may change, rather than being a fixed value. This allows electronic devices to flexibly adjust the start time of image synthesis for each frame, enabling them to delay the start of synthesis in cases of high display latency, thereby reducing display latency, or to start synthesis earlier in cases where display stuttering may occur, thereby avoiding display stuttering.

[0010] In one possible design of the first aspect, after starting to synthesize the first image, the method further includes: waiting for a TE signal, and in response to detecting the TE signal, displaying the first image.

[0011] If the duration of the TE signal wait time for at least one frame (including the first image and possibly consecutive images preceding it) is less than the first duration, it indicates that the time point for completing all processing before display is later, resulting in a shorter TE signal wait time. Consequently, the second time interval can be shorter than the first time interval. This allows for earlier compositing during the processing of the second image, thus avoiding display stuttering.

[0012] If the duration of the TE signal wait time for at least one frame is greater than the first duration, it indicates that the time point for completing all processing before display is earlier, resulting in a longer TE signal wait time and higher display latency. Accordingly, the second time interval can be longer than the first time interval. In this way, the processing of the second image can start compositing later, thereby reducing the display latency.

[0013] In one possible design approach of the first aspect, if the load on the electronic device exceeds a preset load, it indicates that the electronic device is unlikely to efficiently complete the processing of one frame of image. Accordingly, the second time interval can be shorter than the first time interval. In this way, the processing of the second image can begin earlier, thereby avoiding display stuttering in high-load scenarios.

[0014] In one possible design of the first aspect, after displaying the first image in response to the detection of the TE signal, the method further includes: adjusting the offset time based on the waiting time for the TE signal corresponding to at least one frame of image. The third moment is obtained by subtracting the adjusted offset time from the expected display time of the third image.

[0015] This design adjusts the offset time based on the duration of the TE signal wait and further calculates the third moment, so that the synthesis of the third image begins at the third moment. This allows the duration of the TE signal wait for the third image to be controlled within a reasonable time range, thus balancing the issues of high display latency and display stuttering.

[0016] In one possible design approach of the first aspect, the offset time is adjusted according to the duration of the waiting TE signal corresponding to at least one frame of image, including: if the duration of the waiting TE signal corresponding to at least one frame of image is less than the first duration, it indicates that the time point for completing all processing before display is later, resulting in a shorter waiting TE signal duration. Accordingly, the offset time can be increased, so that the synthesis of the third image can start earlier and avoid display stuttering.

[0017] If the duration of waiting for the TE signal for at least one frame of image is greater than the second duration, it indicates that the time point for completing all processing before display is earlier, resulting in a longer waiting time for the TE signal and a higher display delay. Correspondingly, the offset time can be reduced, so that the synthesis of the third image can start later, reducing the display delay.

[0018] In one possible design approach of the first aspect, at least one frame image is a series of consecutive frames, and the duration of waiting for the TE signal corresponding to at least one frame image includes: the average duration of waiting for the TE signal corresponding to the series of consecutive frames.

[0019] By adopting this design approach and adjusting the offset time with reference to the average duration of waiting for the TE signal, random errors can be avoided and the rationality of the adjustment can be improved.

[0020] In one possible design approach of the first aspect, if the duration of the TE signal wait time corresponding to at least one frame of image is greater than the first duration and less than the second duration, it indicates that the display delay is appropriate and display stuttering is less likely to occur. In this case, the offset time does not need to be adjusted, and the second time interval is the same as the first time interval. This helps to maintain the TE signal wait time within a reasonable range.

[0021] In one possible design approach of the first aspect, adjusting the offset time based on the duration of waiting for the TE signal corresponding to at least one frame of image includes: adjusting the offset time based on the duration of waiting for the TE signal corresponding to at least one frame of image when the load of the electronic device does not exceed a preset load.

[0022] By adopting this design approach, the offset time is adjusted based on the waiting time for the TE signal only when the load is not very high. This avoids the offset time being unreasonable due to high load and improves the rationality of the offset time.

[0023] In one possible design of the first aspect, the above method further includes: increasing the offset time according to the load of the electronic device when the load of the electronic device exceeds a preset load.

[0024] By adopting this design method, under high load conditions, the determined offset time can be matched with the load, thus avoiding display stuttering caused by high load.

[0025] In one possible design of the first aspect, the electronic device includes a display driver and an image compositer. Adjusting the offset time based on the duration of a wait TE signal corresponding to at least one frame of image includes: the display driver calculating an adjustment amount for the offset time based on the duration of the wait TE signal corresponding to at least one frame of image; the display driver sending the adjustment amount to the image compositer; and the image compositer adjusting the offset time based on the adjustment amount.

[0026] Using this design approach, the adjustment amount can be calculated by the display driver, and then SurfaceFlinger can quickly determine the third moment based on the adjustment amount, thereby triggering the synthesis of the third image at the third moment, so that the adjusted offset time can take effect in the third image.

[0027] Secondly, this application also provides an electronic device, which includes a System-on-a-Chip (SoC), a display screen, a memory, and one or more processors. The SoC, display screen, memory, and processors are coupled together. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the following steps:

[0028] At the first moment, the synthesis of the first image begins. At the second moment, the synthesis of the second image begins, following the first moment. At the third moment, the synthesis of the third image begins, following the second moment. The first, second, and third images are three consecutive frames, with the second image following the first, and the third image following the second. The frame rate remains constant from the first moment to the third moment, and the first time interval between the first and second moments differs from the second time interval between the second and third moments.

[0029] In one possible design of the first aspect, when computer instructions are executed by a processor, the electronic device performs the following steps: waiting for a TE signal, and in response to detecting the TE signal, displaying a first image. Wherein, if the duration of the TE signal waiting for at least one frame is less than a first duration, a second time interval is less than the first time interval; if the duration of the TE signal waiting for at least one frame is greater than the first duration, the second time interval is greater than the first time interval; and the at least one frame includes the first image.

[0030] In one possible design approach of the first aspect, when the load on the electronic device is higher than a preset load, the second time interval is shorter than the first time interval.

[0031] In one possible design of the first aspect, when the computer instructions are executed by the processor, the electronic device performs the following steps: adjusting the offset time according to the duration of waiting for the TE signal corresponding to at least one frame of image; and subtracting the adjusted offset time from the expected display time of the third image to obtain the third moment.

[0032] In one possible design of the first aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: if the duration of the wait for the TE signal corresponding to at least one frame of image is less than a first duration, the offset time is increased; if the duration of the wait for the TE signal corresponding to at least one frame of image is greater than a second duration, the offset time is decreased.

[0033] In one possible design approach of the first aspect, at least one frame image is a series of consecutive frames, and the duration of waiting for the TE signal corresponding to at least one frame image includes: the average duration of waiting for the TE signal corresponding to the series of consecutive frames.

[0034] In one possible design of the first aspect, if the duration of the waiting TE signal corresponding to at least one frame of image is greater than the first duration and less than the second duration, the offset time is not adjusted, and the second time interval is the same as the first time interval.

[0035] In one possible design of the first aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: adjusting the offset time according to the duration of waiting for the TE signal corresponding to at least one frame of image, provided that the load on the electronic device does not exceed a preset load.

[0036] In one possible design of the first aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: if the load of the electronic device exceeds a preset load, the offset time is increased according to the load of the electronic device.

[0037] In one possible design of the first aspect, the electronic device includes a display driver and an image compositer. When computer instructions are executed by a processor, the electronic device performs the following steps: the display driver calculates an adjustment amount for the offset time based on the duration of the wait time for a TE signal corresponding to at least one frame of image. The display driver sends the adjustment amount to the image compositer. The image compositer adjusts the offset time according to the adjustment amount.

[0038] Thirdly, this application provides a chip system applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method as described in the first aspect and any of its possible design embodiments.

[0039] Fourthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in the first aspect and any possible design thereof.

[0040] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method as described in the first aspect and any possible design thereof.

[0041] Understandably, the beneficial effects that the electronic device of the second aspect, the chip system of the third aspect, the computer storage medium of the fourth aspect, and the computer program product of the fifth aspect can achieve can be referred to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0042] Figure 1 is a schematic diagram of an image display principle provided in an embodiment of this application;

[0043] Figure 2 is a schematic diagram of a display timing provided in an embodiment of this application;

[0044] Figure 3 is a schematic diagram of an image processing flow provided in an embodiment of this application;

[0045] Figure 4 is a schematic diagram of a scenario with high display latency provided in an embodiment of this application;

[0046] Figure 5 is a schematic diagram of a display lag scenario provided by an embodiment of this application;

[0047] Figure 6 is a simplified diagram of an image processing procedure provided in an embodiment of this application;

[0048] Figure 7 is a hardware structure diagram of a mobile phone provided in an embodiment of this application;

[0049] Figure 8 is a hardware and software architecture diagram of a mobile phone provided in an embodiment of this application;

[0050] Figure 9 is a flowchart of an image processing method provided in an embodiment of this application;

[0051] Figure 10 is a schematic diagram illustrating the principle of adjusting the processing timing of a second image according to an embodiment of this application;

[0052] Figure 11 is a schematic diagram of a method for reducing offset time according to an embodiment of this application;

[0053] Figure 12 is a schematic diagram of an embodiment of this application for increasing the offset time;

[0054] Figure 13 is a flowchart of another image processing method provided in an embodiment of this application;

[0055] Figure 14 is a diagram showing the effect of continuously adjusting the offset time according to an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0057] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0058] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] Before introducing the embodiments of this application, the principle of image display will be briefly introduced below with reference to Figures 1 and 2.

[0060] Referring to Figure 1, the electronic device includes a system-on-chip (SoC) and a display screen (also known as the screen end).

[0061] The SoC can integrate key chips such as an application processor (AP) and a baseband processor (BP, also known as a modem). Furthermore, the AP may include modules such as a surface fllinger (SF) and a display driver. The display screen may include a display control module, an image storage unit, and a display panel.

[0062] For example, the display control module may be a display driver integrated circuit (DDIC). It should be noted that the display control module may also be referred to as a display screen controller, driver circuit, etc., and this application embodiment does not specifically limit it in this way.

[0063] For example, the image storage unit can be random access memory (RAM), such as graphics RAM (GRAM).

[0064] The following explanation will use DDIC as an example, where the display control module is a display controller and GRAM is the image storage unit.

[0065] The SoC can perform image processing workflows such as drawing, rendering, and compositing to obtain images. The specific processing flow is shown in Figure 3 and its related description below, and will not be elaborated further here. Then, the SoC can send the image to the display screen for display, a process known as image delivery. For example, the SoC sends (i.e., writes) image data to the GRAM in the display screen, and finally, the DDIC refreshes (i.e. reads) the image data from the GRAM onto the display panel, thus achieving image display.

[0066] The SoC can send the display based on the tear effect (TE) signal. Referring to Figure 2, the TE signal can be a periodic pulse signal. After the DDIC completes refreshing one frame of image on the display panel, it can generate a TE signal, triggering the SoC to write the new frame of image data into the GRAM. For example, after detecting the rising edge of the TE signal, the SoC can send the display to the screen.

[0067] Referring to Figure 3, the processing flow of one frame of image includes: drawing, rendering, compositing, preparation before display, and display. For example, the processing flow of image A includes drawing image A, rendering image A, compositing image A, preparation before display of image A, and display of image A, ultimately displaying image A. The processing flows for images B, C, and D are similar, and will not be illustrated further here.

[0068] Specifically, a System-on-a-Chip (SoC) (such as an application, APP) can trigger the drawing of a frame of image based on the APP's VSYNC signal (often denoted as VSYNC-APP). As shown in the diagram, drawing of a frame of image begins with each pulse of VSYNC-APP. Then, the SoC (such as a Graphics Processing Unit, GPU) can perform rendering. Similarly, the SoC (such as SurfaceFlinger) can trigger the compositing of a frame of image based on VSYNC-SF. As shown in the diagram, compositing of a frame of image begins with each pulse of VSYNC-SF, resulting in an image. After the drawing, rendering, and compositing processes are completed, the SoC (such as a display driver) can perform pre-display preparations, such as initializing the hardware image submission registers and initializing the image submission timing. Subsequently, after detecting the TE signal generated by the display screen, the SoC (such as the display driver) can send the composited image to the display, and the display screen can finally display the image.

[0069] Among them, SoC (such as SurfaceFlinger) can simulate VSYNC-SF based on historically detected TE signals, so that the synthesis based on VSYNC-SF can be started, ensuring that the synthesis, pre-display preparation and other processing are completed before the next TE signal arrives, and the new frame image is ready.

[0070] This application provides an exemplary implementation of a specific method for simulating VSYNC-SF:

[0071] The SoC can predict the arrival time of subsequent TE signals based on historically detected TE signals, thus determining the Expected Display Time (EPT). EPT is commonly used in Variable Refresh Rate (VRR) technology. In VRR, by calculating the EPT, the image synthesizer can anticipate the start time of synthesis, ensuring a new frame is ready before the EPT (which can be understood as before the next TE signal arrives). For example, the SoC can push the EPT forward by an offset time to obtain a pulse moment of VSYNC-SF, i.e., the start time of synthesis. For instance, after estimating the EPT for image B in Figure 3 as Tb, SurfaceFlinger calculates Tb minus the offset time Δtb to obtain the start time point SFb for synthesizing image B. The offset time is reserved for completing synthesis and preparation before EPT. The SoC starts synthesis when the VSYNC-SF pulse arrives and completes synthesis and preparation within the offset time, ensuring a new frame is ready before the EPT. In this way, when the SoC detects the arrival of the TE signal, it can send a prepared image for display.

[0072] The SoC (such as an application) can also simulate the VSYNC signal of the application based on VSYNC-SF (often referred to as VSYNC-APP), so that drawing can start based on VSYNC-APP, ensuring timely completion of drawing and rendering. Simulating VSYNC-APP based on VSYNC-SF is not the focus of this application's embodiments, and will not be elaborated upon further.

[0073] In the above simulation of VSYNC-SF, the offset time is usually tied to the frame rate. For example, at a frame rate of 120 FPS, the offset time is 10 ms; at 60 FPS, it's 18 ms; and at 30 FPS, it's 35 ms. Referring again to Figure 3, taking a frame rate of 60 FPS as an example, Δta = Δtb = Δtc = Δtd = 18 ms. Therefore, as long as the frame rate remains constant, the expected time interval between two adjacent frames is a fixed value. Correspondingly, the time interval at which two adjacent frames begin to be combined is also fixed, as shown in Figure 3, where the time interval between SFb and SFc is equal to the time interval between SFc and SFd.

[0074] However, the VSYNC-SF trigger for starting synthesis, simulated based on an offset time tied to the frame rate, may result in high display latency or display stuttering. Display latency refers to the time from the start of rendering a frame to the display of the image. Referring to Figure 3, taking image A as an example, the display latency of image A is the time from the start of rendering image A to the end of its display.

[0075] The following sections explain the two scenarios: high display latency and display stuttering.

[0076] In case a, the display delay is relatively high.

[0077] Referring to Figure 4, shortly after completing the pre-display preparation for Figure C (denoted as t1c in the figure), the SoC detects the TE signal (denoted as TEc in the figure) and executes the display of Figure C. Similarly, shortly after completing the pre-display preparation for Figure D (denoted as t1d in the figure), the SoC detects the TE signal (denoted as TEd in the figure) and executes the display of Figure D. However, after completing the pre-display preparation for Figure E, a longer time interval (denoted as t1e in the figure), such as t1e as high as 10ms, is required before the SoC detects the TE signal (denoted as TEe in the figure) and executes the display of Figure E. In other words, the drawing of Figure E begins too early, resulting in a longer waiting time after completing the pre-display preparation for Figure E before it can be displayed. In this case, it indicates that the display delay of Figure E is high.

[0078] In scenarios requiring immediate response, such as stylus input or skill activation in games, high latency can lead to untimely responses and a poor user experience. For example, with styluses, high latency might mean the writing takes a considerable amount of time to appear on the screen, resulting in poor responsiveness. Similarly, in games, high latency can cause delays in displaying skill effects, such as hit detection, after the user clicks the skill button, leading to a poor user experience.

[0079] It is evident that while the offset time tied to the frame rate can ensure that a new frame of image is prepared before the next TE signal arrives, it may result in high latency in sending certain images to the display, affecting the user experience.

[0080] Case b: Display lag.

[0081] Referring to Figure 5, after completing the preparations for displaying image D, the SoC detects the TE signal (denoted as TEd in the figure) and executes the display of image D. However, because the rendering of image F takes a relatively long time, when the SoC detects the next TE signal after TEd (denoted as TEf in the figure), the SoC is still rendering image F and has not completed the preparations for displaying image F. Therefore, the display of image F cannot be executed at TEf. Consequently, the display will show another frame of image D. Subsequently, before the SoC detects the next TE signal after TEf (denoted as TEf' in the figure), the SoC has completed the preparations for displaying image F. Therefore, the display of image F can be executed upon detecting TEf'. In other words, image D is displayed for two consecutive frames, resulting in display stuttering.

[0082] In scenarios where smooth display is crucial, such as gaming and video playback, display stuttering can significantly impact the user's viewing experience.

[0083] As can be seen, while the offset time tied to the frame rate can meet the time requirements of processing one frame of image in most cases, under some relatively high load conditions, the SoC may not be able to complete image processing in time, making it impossible to prepare a new frame of image before the TE signal arrives, thus causing display stuttering. For example, under high GPU load, the GPU may not be able to complete rendering in time. Furthermore, after completing compositing, the display driver may not be able to perform pre-display preparation based on the rendering and compositing results, making it impossible to prepare a new frame of image before the TE signal arrives.

[0084] Based on the above problems, this application proposes an image processing method in which the SoC can dynamically adjust the offset time based on the duration of waiting for the TE signal in the processing flow of historical images, so that the duration of waiting for the TE signal after preparing a new frame image can be controlled within a range that is greater than a first duration and less than a second duration, such as the first duration being 3ms and the second duration being 5ms.

[0085] Referring to Figure 6, in the processing flow of a single frame of image, after drawing, rendering, compositing, and preparations before display are all completed, the system can begin waiting for the TE signal. Once the TE signal is received, display can proceed. The waiting time for the TE signal is t1, as shown in t1c, t1d, and t1e in Figure 4 above. A larger t1 indicates an earlier start time for drawing, while a smaller t1 indicates a later start time.

[0086] When t1 is greater than or equal to the second duration, the SoC can reduce the offset time, which is equivalent to reducing the time difference between VSYNC-SF and the expected display time, reserving less time for image processing. This can shorten the waiting time for the TE signal in the subsequent image processing flow, making the waiting time for the TE signal less than the second duration, and reducing the display delay.

[0087] When t1 is less than or equal to the first duration, the SoC can increase the offset time, which is equivalent to increasing the time difference between VSYNC-SF and the expected display time, reserving more time for image processing. This allows the waiting time for the TE signal in subsequent image processing to be increased, making the waiting time for the TE signal greater than the first duration. This ensures that a new frame of image can be prepared before the TE signal arrives, avoiding display stuttering.

[0088] In other words, even if the frame rate remains constant, meaning the time interval between the expected display times of two adjacent frames is a fixed value, the time interval at which two adjacent frames begin to be combined will no longer be fixed due to the adjustment of the offset time.

[0089] At this point, it should be noted that adjusting the offset time is equivalent to adjusting VSYNC-SF. As mentioned earlier, VSYNC-APP is based on VSYNC-SF simulation. Therefore, adjusting the offset time is equivalent to adjusting the time reserved for one frame of image, that is, the time from the start of drawing triggered by VSYNC-APP to the time of display.

[0090] Furthermore, reducing the offset time: the angle of the TE signal is equivalent to moving VSYNC-SF closer to the expected display time, thus increasing the corresponding VSYNC-SF time. Conversely, the angle of the VSYNC-SF is equivalent to moving the expected display time closer to VSYNC-SF, thus decreasing the expected display time. Increasing the offset time: the angle of the TE signal is equivalent to moving VSYNC-SF further away from the expected display time, thus decreasing the corresponding VSYNC-SF time. Conversely, the angle of the VSYNC-SF is equivalent to moving the expected display time further away from VSYNC-SF, thus increasing the expected display time. Therefore, in other words, adjusting the offset time can also be considered adjusting the expected display time. Decreasing the offset time decreases the expected display time, and increasing the offset time increases the expected display time. However, it is important to clarify that the expected display time only decreases or increases relative to VSYNC-SF; the absolute value of the expected display time remains unchanged. Only the offset time and the VSYNC-SF simulated based on the offset time change.

[0091] For example, the electronic devices in this application embodiment may be mobile phones, tablets, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices, etc., that have image processing and display requirements. This application embodiment does not impose any special limitations on the specific form of the electronic device.

[0092] Referring to Figure 7, which is a hardware structure diagram of an electronic device. As shown in Figure 7, taking a mobile phone as an example, the electronic device may include a processor 210, an external memory interface 220, an internal memory (RAM) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0093] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0094] The processor 210 may include one or more processing units, such as an application processor (AP), a modem, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural network processing unit (NPU).

[0095] Different processing units can be independent devices or integrated into one or more processors. For example, an AP can integrate a CPU, GPU, etc.

[0096] In some embodiments, the processor 210 can execute an image processing method by running instructions stored in the internal memory 221.

[0097] In addition, one or more processing units in the aforementioned processor 210, as well as other components of the mobile phone (such as memory, input / output interfaces, etc.), can be integrated into the SoC.

[0098] The charging management module 240 receives charging input from the charger. The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, display screen 294, camera 293, and wireless communication module 260, etc. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status.

[0099] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor.

[0100] Mobile phones can achieve display functions through GPU, display screen 294, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering.

[0101] Mobile phones can achieve camera functions through the camera 293, ISP, video codec, GPU, display 294, application processor AP, neural network processor NPU, etc.

[0102] Mobile phones can perform audio functions, such as music playback and recording, through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, and an application processor.

[0103] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, a mobile phone may include one or more displays screens 294.

[0104] The display screen 294 also includes driving circuitry and a storage unit (such as GRAM, which will be used as an example below). The GRAM can be used to store image data of the image to be displayed.

[0105] For details on the functions and working principles of the display panel, DDIC, and GRAM, please refer to Figure 1 above and its related descriptions; they will not be repeated here.

[0106] The software system of the aforementioned electronic device (such as the software system of a SoC) can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture of Android. TM Taking a system as an example, this section illustrates the software structure of an electronic device. A layered architecture divides the software system of an electronic device into several layers, each with a clear role and function, and the layers communicate with each other through software interfaces.

[0107] Referring to Figure 8, taking a mobile phone as an example, the hardware and software architecture of an electronic device can include an application layer, an application framework layer, a native layer, a hardware abstraction layer (HAL), a kernel layer, and a hardware layer. Among them, the application layer, application framework layer, native layer, and hardware abstraction layer belong to user space, while the kernel layer and hardware layer belong to kernel space.

[0108] The application layer can accommodate various applications such as email, video player, memo, alarm clock, settings, and games. When these applications run in the foreground, they can all be processed by the electronic device to obtain and display images.

[0109] The application framework layer provides the application programming interface (API) and programming framework for the application layer. The application framework layer includes some predefined functions.

[0110] The application framework layer can include system services such as the window manager, activity manager, and resource manager.

[0111] The local layer includes SurfaceFlinger. SurfaceFlinger can be used for layer compositing, backlighting, screen on / off control, etc.

[0112] For example, SurfaceFlinger can assign compositing tasks to layers drawn by the application, distributing these tasks to the Hardware Composer (HWC) in the Hardware Abstraction Layer or the GPU. For instance, if the compositing task includes tasks such as rounded corner clipping, it can be assigned to the GPU for image compositing; if the compositing task only includes simple layer overlay tasks, it can be assigned to the HWC. This allows for control over image compositing.

[0113] SurfaceFlinger can also calculate the expected display time and simulate VSYNC-SF based on the expected display time and offset time, thereby triggering image synthesis.

[0114] Furthermore, SurfaceFlinger can include the following modules: a dynamic time adjustment module, a VSYNC-SF module, and a layer compositing module. The dynamic time adjustment module is used to adjust the offset time. The VSYNC-SF module is used to simulate VSYNC-SF. The layer compositing module is used for image compositing.

[0115] The Hardware Abstraction Layer (HWC) can encapsulate kernel-level drivers, providing calling interfaces to higher layers. The HWC can include HWC. HWC can be used for image compositing and for submitting the composited image data to the display driver. It should be noted that the image data obtained by the GPU during image compositing is also submitted to the display driver via HWC.

[0116] HWC can also be used to transfer underlying data to SurfaceFlinger, which is then used by SurfaceFlinger to adjust the offset time.

[0117] In some embodiments, HWC can transmit the waiting time t1 for the TE signal during the processing of each frame of image to SurfaceFlinger.

[0118] In other embodiments, HWC can transmit the adjustment amount of the offset time (such as increasing by 2ms, decreasing by 3ms, etc.) to SurfaceFlinger.

[0119] Furthermore, HWC can include the following modules: a parameter transfer module and an atomic submission module. The parameter transfer module can be used to transfer the aforementioned duration t1 or offset time adjustment to SurfaceFlinger. The atomic submission module can be used to submit the synthesized image to the display driver.

[0120] The kernel layer (also known as the driver layer) includes drivers that enable the hardware to function. Specifically, the kernel layer includes display control commit modules (such as CRT commit) and display drivers (such as the LCD Driver).

[0121] The display control submission module can be used for information transfer between the upper layer and the driver layer to achieve display control. For example, the display control submission module can transmit an image from itself to the display driver.

[0122] The display driver can be used to send the synthesized image to a display screen for display. Specifically, the display driver can perform the display sending after detecting the TE signal.

[0123] The display driver can also be used to transmit data for adjusting the offset time to the upper layer, such as the duration t4 or the adjustment amount of the offset time.

[0124] The hardware layer may include hardware such as CPU, GPU, and display screen (further including display panel, DDIC, GRAM, etc.).

[0125] It is understandable that the division of hardware and software modules in the descriptions of electronic devices in Figures 7 and 8 above are exemplary and are not limited to these in practice.

[0126] The image processing method provided in this application embodiment can be executed in an electronic device having the above-described hardware and software structure. Specifically, in a SoC, the offset time can be adjusted based on the duration of waiting for the TE signal corresponding to at least one frame of image, thereby reducing the display latency when the display latency is high, or avoiding display stuttering when the duration of waiting for the TE signal is very short (which easily leads to display stuttering).

[0127] In some embodiments, the adjustment amount for the offset time can be determined by the display driver, and SurfaceFlinger can then adjust the offset time based on this adjustment amount. Referring to Figure 9, the image processing method includes the following steps:

[0128] S900, the display screen generates a TE signal.

[0129] For example, the display can generate TE signals at frequencies such as 360Hz, 180Hz, 90Hz, and 60Hz to trigger the SoC to send images at the corresponding frequencies. For instance, if the display generates a 120Hz TE signal, it can trigger the SoC to send 120 frames of images to the display at 120Hz, or 120 frames per second.

[0130] It should be noted that S900 is a continuous operation, and the display can continuously generate TE signals, rather than only generating TE signals before S901.

[0131] S901, in response to VSYNC-APP1, APP performs layer drawing of the first image to obtain a layer with the first content.

[0132] The first image is a frame of image that the app will display. Taking a game app as an example, the first image could be a frame of game footage.

[0133] For example, the app can calculate the trigger time of VSYNC-APP1 based on VSYNC-SF (hereinafter VSYNC-SF1). For instance, the app can subtract the offset between VSYNC-SF and VSYNC-APP, such as 3ms, from the trigger time of VSYNC-SF1 to obtain the trigger time of VSYNC-APP1. Subsequently, at the trigger time of VSYNC-APP1, VSYNC-APP1 can be generated, thereby triggering the layer drawing of the first image.

[0134] It's understandable that rendering can be performed after layer drawing. For example, an app can call the GPU to perform rendering.

[0135] S902, the APP sends the layer with the first content to SurfaceFlinger.

[0136] S903, in response to VSYNC-SF1, SurfaceFlinger performs the synthesis of the first image to obtain the first image.

[0137] For example, SurfaceFlinger can simulate VSYNC-SF1 based on historically detected TE signals. For instance, it can estimate the expected display time of the first image based on the historically detected TE signals, and then subtract the offset time (which can be an offset time bound to the frame rate or an adjusted offset time) from the expected display time of the first image to obtain the trigger time of VSYNC-SF1. Subsequently, at the trigger time of VSYNC-SF1, VSYNC-SF1 can be generated, thereby triggering the synthesis of the first image.

[0138] For ease of explanation, the trigger time of VSYNC-SF1 can also be referred to as the first moment. For example, if the first image is Figure I in Figure 11 or Figure 12 below, then the first moment is SFi.

[0139] It is understandable that SurfaceFlinger can assign compositing tasks to the GPU and / or HWC, and complete the compositing through the GPU and / or HWC.

[0140] S904, SurfaceFlinger sends the first image to the display driver.

[0141] For example, SurfaceFlinger can send images to the display driver via HWC.

[0142] S905, The display driver performs pre-display preparation for the first image.

[0143] S906, Display driver is waiting for TE signal.

[0144] After completing the pre-display preparations, the conditions for display are met, and the display driver can enter a state of waiting for the TE signal, such as the wait completion event state, waiting for the arrival of the TE signal.

[0145] S907, In response to the detection of the TE signal, a first image is sent to the display screen.

[0146] Subsequently, the display screen can show the first image.

[0147] In this embodiment, after the first image is displayed, the VSYNC-SF can be adjusted on the SoC side via the following S908-S910.

[0148] S908, the display driver calculates the adjustment amount of the offset time based on the waiting time t1 of the TE signal.

[0149] The adjustment amounts include: increasing the duration, such as increasing by 1ms or 2ms; or decreasing the duration, such as decreasing by 1ms or 2ms.

[0150] In one specific implementation, the display driver can calculate the adjustment amount of the offset time based on the average value of the waiting time t1 for the TE signal corresponding to multiple consecutive frames. Specifically, if the average value is greater than or equal to a second duration, the display driver decreases the offset time. If the average value is less than or equal to a first duration, the display driver increases the offset time.

[0151] Furthermore, the display driver can increase or decrease the offset time in fixed steps, such as increasing or decreasing by 1ms each time. In this way, the offset time can be gradually adjusted so that after multiple adjustments, the waiting time for TE can be adjusted to a range greater than the first time and less than the second time.

[0152] Alternatively, the display driver can increase or decrease the offset time to varying degrees based on the average value. Specifically, when the average value is greater than or equal to the second duration, a larger average value results in a greater reduction in offset time, and a smaller average value results in a smaller reduction in offset time. When the average value is less than or equal to the first duration, a smaller average value results in a greater increase in offset time, and a larger average value results in a smaller increase in offset time.

[0153] Furthermore, if the average value is greater than the first duration and less than the second duration, the offset time does not need to be adjusted. In this case, the display driver does not need to calculate the adjustment amount, or the calculated adjustment amount is 0. Also, without adjusting the offset time, the time interval between the start of compositing between two adjacent frames is fixed.

[0154] Of course, the display driver can also calculate the adjustment amount of the offset time based on the waiting time t1 of the TE signal corresponding to the current first image. Wherein, if the duration t1 is greater than or equal to the second duration, the offset time is decreased; if the duration t1 is less than or equal to the first duration, the offset time is increased. This application does not specifically limit this.

[0155] S909, the display driver sends adjustment values ​​to SurfaceFlinger.

[0156] S910 and SurfaceFlinger determine the adjusted offset time based on the adjustment amount, and then determine the trigger time of VSYNC-SF2 based on the adjusted offset time.

[0157] VSYNC-SF2 is used to trigger the synthesis of the third image.

[0158] It should be noted that the third image is the image following the first image, and typically, the third image is the image one frame after the first image; this one-frame interval can be called the second image. Furthermore, for ease of description, the time at which the second image begins to be synthesized can be called the second moment. If the second image is image J in Figures 11 and 12 below, then the second moment can be SFj.

[0159] Referring to Figure 10, after the H image is displayed, the I image has already begun processing. There is no need to determine the timing for starting to draw the I image or the timing for starting to synthesize the I image. At this point, the J image has not yet begun processing. The timing for starting to synthesize the J image can be determined based on the adjusted offset time (e.g., the trigger time of VSYNC-SF2), and then the timing for starting to draw the J image can be determined based on the timing for starting to synthesize the J image (e.g., the trigger time of VSYNC-APP2 in S912 below). That is, the H image is the first image, the J image is the third image, and the I image is separated from the H image and the J image.

[0160] Referring again to Figure 10, after image I is displayed, image J has already begun processing. There's no need to determine when to start drawing or compositing image J. At this point, image K has not yet begun processing. The timing for starting image K compositing can be determined based on the adjusted offset time (e.g., the trigger time of VSYNC-SF2), and then the timing for starting image K drawing can be determined based on the timing for starting image K compositing (e.g., the trigger time of VSYNC-APP2 in S912 below). That is, image I is the first image, image K is the third image, and image J is the interval between image I and image K.

[0161] In this embodiment, SurfaceFlinger can obtain the adjustment amount of the offset time, and thus increase or decrease the current offset time to obtain the adjusted offset time. For example, if the current offset time is 18ms and the adjustment amount is a decrease of 1ms, the adjusted offset time is 17ms. As another example, if the current offset time is 10ms and the adjustment amount is an increase of 1ms, the adjusted offset time is 11ms.

[0162] After obtaining the adjusted offset time, SurfaceFlinger can subtract the adjusted offset time from the expected display time of the third image to obtain the VSYNC-SF2 trigger time. In other words, the VSYNC-SF2 trigger time is not based on a fixed offset time bound to the frame rate, but on a dynamically adjusted offset time, and this dynamically adjusted offset time is related to the duration t1 of waiting for the TE signal corresponding to one frame or multiple consecutive frames. Thus, the VSYNC-SF2 trigger time can be adjusted in different directions depending on the different durations t1 of waiting for the TE signal corresponding to one frame or multiple consecutive frames, as shown below:

[0163] In scenario one, if the waiting time t1 for the TE signal in one frame or multiple consecutive frames is relatively long (greater than or equal to the second duration), SurfaceFlinger can shift the trigger time of VSYNC-SF2 later, making the trigger time of VSYNC-SF2 closer to the expected display time of the third image.

[0164] Taking the first image as I and the third image as K, with a duration of 5ms as an example (see Figure 11), the waiting time for the TE signal corresponding to image H is 8ms, and the waiting time for the TE signal corresponding to image I is 10ms. The average of the two is 9ms, which is greater than 5ms. The display driver determines the adjustment amount of the offset time to be reduced by 2ms. SurfaceFlinger can subtract 2ms from the current offset time, i.e., the time interval between the start of image J synthesis (VSYNC-SFj, abbreviated as SFj in the figure) and the expected display time Tj of image J, which is 18ms, to obtain the adjusted offset time of 16ms. SurfaceFlinger subtracts 16ms from the expected display time Tk of image K to obtain the start of image K synthesis (VSYNC-SFk, abbreviated as SFk in the figure), i.e., the trigger time of VSYNC-SF2 is SFk. If calculated according to the offset time of 18ms, the start of image K synthesis is as shown in SFk', located to the left of SFk. This means that SFk has been moved back, closer to the expected delivery time Tk of the K-chart.

[0165] In scenario two, if the waiting time t1 for the TE signal in one frame or multiple consecutive frames is short (less than or equal to the first duration), SurfaceFlinger can move the trigger time of VSYNC-SF2 forward, making the trigger time of VSYNC-SF2 further away from the expected display time of the third image.

[0166] Taking the first image as I and the third image as K, with a duration of 3ms as an example (see Figure 12), the waiting time for the TE signal corresponding to image H is 3ms, and the waiting time for the TE signal corresponding to image I is 2ms. The average of the two is 2.5ms, which is less than 3ms. The display driver determines the adjustment amount of the offset time to increase by 2ms. SurfaceFlinger can increase the current offset time, i.e., the time interval between the start of image J synthesis (VSYNC-SFj, abbreviated as SFj in the figure) and the expected display time Tj of image J (18ms), by 2ms, to obtain the adjusted offset time of 20ms. SurfaceFlinger subtracts 20ms from the expected display time Tk of image K to obtain the start of image K synthesis (VSYNC-SFk, abbreviated as SFk in the figure), i.e., the trigger time of VSYNC-SF2 is SFk. If calculated according to an offset time of 18ms, the start of image K synthesis is as shown in SFk', located to the right of SFk. This means that SFk has moved forward, further away from the expected delivery time Tk of the K-chart.

[0167] Furthermore, prior to S910, SurfaceFlinger could also verify the legality of the adjustment amount to avoid adjusting the offset time to an unreasonable duration range. For example, if the absolute value of the adjustment amount was greater than or equal to the current offset time, it indicated that the adjustment was too large and therefore invalid. In one specific implementation, SurfaceFlinger could also verify the legality of the adjustment amount in conjunction with the frame rate. Different frame rates correspond to different reasonable duration ranges. SurfaceFlinger could verify whether the adjusted offset time obtained based on the adjustment amount was within the reasonable duration range corresponding to the current frame rate. If not, it was invalid; otherwise, it was valid.

[0168] Next, on the SoC side, VSYNC-APP can also be adjusted via the following S911-S912.

[0169] S911, SurfaceFlinger sends the VSYNC-SF2 trigger time to the APP.

[0170] S912, APP calculates the trigger time of VSYNC-APP2 based on the trigger time of VSYNC-SF2.

[0171] VSYNC-SF2 is used to trigger the drawing of the third image.

[0172] For example, the APP can subtract the offset between VSYNC-SF and VSYNC-APP, such as 3ms, from the trigger time of VSYNC-SF2 to obtain the trigger time of VSYNC-APP2.

[0173] For example, subtracting 3ms from SFk, the timing of starting K-graph synthesis in Figures 11 and 12, yields the timing of starting K-graph drawing: VSYNC-APPk (abbreviated as Pk in the figure), which is the trigger time of VSYNC-APP2.

[0174] It is understandable that after the aforementioned S908-S910, the trigger time of VSYNC-APP2 is shifted forward or backward, and the trigger time of VSYNC-APP2 determined accordingly will also be shifted forward or backward. Thus, the total time reserved for third-party image processing (i.e., the display delay) will be different.

[0175] For example, the time reserved for Pk to Tk in Figure 11 for K-graph processing is obviously shorter than the time between Pk' (e.g., SFk' minus 3ms) and Tk, which helps to reduce the display delay.

[0176] For example, the time reserved for processing Pk to Tk in Figure 12 is obviously longer than the time between Pk' (e.g., SFk' minus 3ms) and Tk, which helps to avoid display stuttering.

[0177] Subsequently, on the SoC side, the processing of the third image can be performed according to the trigger time of VSYNC-SF2 and the trigger time of VSYNC-APP2 determined above, as shown in S913-S919 below:

[0178] S913, In response to VSYNC-APP2, the APP performs layer drawing of the third image, resulting in a layer with the second content.

[0179] After determining the trigger time of VSYNC-APP2 (also known as the third moment), VSYNC-APP2 will be generated, such as a pulse signal, upon its arrival. The APP can detect VSYNC-APP2. After detecting VSYNC-APP2, the APP can begin drawing the layer of the third image. Thus, the APP can begin drawing the layer of the third image at the trigger time of VSYNC-APP2. For example, in Pk in Figures 11 and 12, the APP can begin drawing the layer of image K.

[0180] S914, the APP sends a layer with second content to SurfaceFlinger.

[0181] S915, in response to VSYNC-SF2, SurfaceFlinger performs the synthesis of the third image to obtain the third image.

[0182] After determining the trigger time of VSYNC-SF2, a pulse signal (VSYNC-SF2) will be generated upon its arrival, which SurfaceFlinger can detect. Upon detecting VSYNC-SF2, SurfaceFlinger can then begin the synthesis of the third image. Thus, SurfaceFlinger can begin the synthesis of the third image at the trigger time of VSYNC-SF2. For example, in SFk in Figures 11 and 12, SurfaceFlinger can begin the synthesis of the K-image.

[0183] S916, SurfaceFlinger sends a third image to the display driver.

[0184] S917, The display driver performs pre-display preparation for the third image.

[0185] S918, Display driver is waiting for TE signal.

[0186] By reducing the offset time, the total time reserved for third image processing is shortened. Consequently, the display driver waits for the TE signal for a shorter period, thus reducing the display latency. For example, in Figure 11, the wait time for the TE signal corresponding to graph K is 6ms, which is shorter than the wait time of 8ms for graph H, 10ms for graph I, and 7ms for graph J. This demonstrates that the display latency of graph K is reduced.

[0187] By increasing the offset time, the total time reserved for third-image processing becomes longer. Consequently, the display driver waits for the TE signal for a longer period, thus preventing display stuttering. For example, in Figure 12, the wait time for the TE signal corresponding to image K is 4ms, which is longer than the wait time of 3ms for image H, 2ms for image I, and 3ms for image J. This makes display stuttering less likely to occur.

[0188] S919, In ​​response to the detection of the TE signal, the display driver sends a third image to the display screen.

[0189] For the parts not detailed in S913-S919, please refer to the explanations in S901-S907 above, which will not be repeated here.

[0190] Using the embodiment shown in Figure 9, the display driver can determine the adjustment amount of the offset time based on the duration of waiting for the TE signal. In this way, SurfaceFlinger can quickly determine the trigger time of VSYNC-SF2 based on the adjustment amount, and start the synthesis of the third image at the corresponding trigger time, so that the adjusted offset time can take effect in the third image.

[0191] In other embodiments, the adjustment amount for the offset time can be determined by SurfaceFlinger, and the offset time can be adjusted based on the adjustment amount. Referring to Figure 13, the image processing method includes the following steps:

[0192] S900-S907 execute the drawing, compositing, and display of the first image. For details, please refer to the previous description, which will not be repeated here.

[0193] S1301, the duration t1 for the display driver to send the waiting TE signal corresponding to the first image to SurfaceFlinger.

[0194] In this embodiment, the display driver sends the duration t1 to SurfaceFlinger, which then calculates the adjustment amount for the offset time.

[0195] In this embodiment, the display driver simply sends the duration t1 to the SurfaceFlinger. The SurfaceFlinger can only calculate the adjustment amount of the offset time based on the duration t1 after being woken up, and thus calculate the adjusted offset time. Therefore, in this embodiment, the SoC will still perform the processing of the third image based on the current offset time, as shown in S1302-S1308 below:

[0196] S1302, In response to VSYNC-APP2', the APP performs the drawing of the third image layer, resulting in a layer with the second content.

[0197] VSYNC-APP2' is used to trigger the drawing of the third image. For example, if the third image is the K image in Figure 11 or Figure 12, then VSYNC-APP2' is Pk'.

[0198] S1303, The APP sends a layer with second content to SurfaceFlinger.

[0199] S1304, in response to VSYNC-SF2', SurfaceFlinger performs the synthesis of the third image to obtain the third image.

[0200] VSYNC-SF2' is used to trigger the synthesis of the third image. For example, if the third image is the K image in Figure 11 or Figure 12, then VSYNC-SF2' is SFk'.

[0201] S1305, SurfaceFlinger sends a third image to the display driver.

[0202] S1306, The display driver performs pre-display preparation for the third image.

[0203] S1307, Display driver is waiting for TE signal.

[0204] S1308, In response to the detection of the TE signal, a third image is sent to the display screen.

[0205] For the parts not detailed in S1302-S1308, please refer to the introduction of S913-S919 above, which will not be repeated here.

[0206] It is understandable that after S1308, the display driver will also send the waiting time t1 for the TE signal corresponding to the third image to SurfaceFlinger, so that SurfaceFlinger can adjust the offset time.

[0207] In S1304 above, SurfaceFlinger is awakened. Thereafter, SurfaceFlinger can calculate the adjustment amount of the offset time based on the duration t1, thereby calculating the adjusted offset time for processing the fourth image. The fourth image is the next frame after the third image. Specifically, see S1309-S1312 below:

[0208] S1309, SurfaceFlinger calculates the adjustment amount of the offset time based on the waiting time t1 for the TE signal.

[0209] For details on the specific implementation of the calculation adjustment amount, please refer to the introduction of S908 above, which will not be repeated here.

[0210] S1310 and SurfaceFlinger determine the adjusted offset time based on the adjustment amount, and then determine the trigger time of VSYNC-SF3 based on the adjusted offset time.

[0211] VSYNC-SF3 is used to trigger the synthesis of the fourth image.

[0212] The principle by which SurfaceFlinger determines the trigger time of VSYNC-SF3 is the same as the principle by which it determines the trigger time of VSYNC-SF2 in the previous S910. For details, please refer to the introduction of S910. It will not be repeated here.

[0213] S1311, SurfaceFlinger sends the trigger time of VSYNC-SF3 to the APP.

[0214] S1312, The APP calculates the trigger time of VSYNC-APP3 based on the trigger time of VSYNC-SF3.

[0215] VSYNC-APP3 is used to trigger the synthesis of the fourth image.

[0216] The principle by which the APP determines the trigger time of VSYNC-APP3 is the same as the principle by which the APP determines the trigger time of VSYNC-APP2 in the previous S912. For details, please refer to the introduction of S912 in the previous text. It will not be repeated here.

[0217] Subsequently, on the SoC side, the processing of the fourth image can be performed according to the trigger time of VSYNC-SF3 and the trigger time of VSYNC-APP3 determined above, as shown in S1313-S1319 below:

[0218] S1313, In response to VSYNC-APP3, the APP performs the drawing of the fourth image layer, resulting in a layer with the third content.

[0219] S1314. The APP sends a layer with third-party content to SurfaceFlinger.

[0220] S1315, In response to VSYNC-SF3, SurfaceFlinger performs the synthesis of the fourth image to obtain the fourth image.

[0221] In addition, in S1315, SurfaceFlinger is woken up again. After that, SurfaceFlinger can calculate the new adjustment amount of the offset time based on the new duration t1, such as the waiting time t1 for the TE signal corresponding to the third image, and thus calculate the new adjusted offset time for processing subsequent images.

[0222] S1316, SurfaceFlinger sends a fourth image to the display driver.

[0223] S1317, The display driver performs pre-display preparation for the fourth image.

[0224] S1318, Display driver is waiting for TE signal.

[0225] S1319. In response to the detection of the TE signal, the display driver sends a fourth image to the display screen.

[0226] Regarding the implementation principle of processing the fourth image in S1313-S1319, it is similar to the implementation principle of processing the third image in S913-S919 mentioned above. For details, please refer to the introduction of S913-S919 mentioned above, which will not be repeated here.

[0227] Using the embodiment shown in Figure 13, the display driver sends the waiting time t1 for the TE signal to SurfaceFlinger. After being woken up, SurfaceFlinger calculates the adjustment amount of the offset time. In this way, SurfaceFlinger can determine a suitable adjustment amount based on the current offset time, so that the determined adjustment amount is within a reasonable range, thereby obtaining a more accurate offset time for subsequent image processing.

[0228] In some embodiments, the SoC can also take into account load factors such as CPU frequency, GPU frequency, and device temperature, and increase the offset time when the load exceeds the load threshold to adapt to the high load requirements.

[0229] For example, when the CPU frequency is higher than a first frequency value, the GPU frequency is higher than a second frequency value, and / or the device temperature is higher than a first temperature, the SoC increases the offset time so that sufficient time is reserved for image processing under high load.

[0230] In other embodiments, in the embodiments of Figure 9 or Figure 13 above, the SoC can directly increase the offset time when the load exceeds the load threshold, and adjust the offset time based on the waiting time t1 for the TE signal when the load does not exceed the load threshold.

[0231] In one specific implementation, corresponding to the embodiment in Figure 9 above, before S908, the display driver can detect whether the load exceeds a load threshold. If the load threshold is exceeded, the display driver determines an adjustment amount that matches the load. If the load threshold is not exceeded, the display driver determines the adjustment amount according to S909.

[0232] In another specific implementation, corresponding to the embodiment in Figure 13 above, before S1309, SurfaceFlinger can detect whether the load exceeds a load threshold. If the load threshold is exceeded, SurfaceFlinger determines an adjustment amount that matches the load. If the load threshold is not exceeded, SurfaceFlinger determines the adjustment amount according to S1309.

[0233] The above embodiments mainly describe the adjustment process of the offset time. In actual implementation, the SoC can dynamically adjust the offset time for multiple consecutive frames, thereby controlling the waiting time for the TE signal within the range of the first duration to the second duration. Referring to Figure 14, as the offset time gradually decreases, the waiting time t1 for the TE signal (abbreviated as waiting time in the figure) also decreases, such as from greater than the second duration to less than the first duration. Then, as the offset time gradually increases, the waiting time t1 for the TE signal also increases, such as from less than the first duration to greater than the second duration. Then, after increasing the offset time to a certain value, the waiting time t1 for the TE signal stabilizes between the first duration and the second duration, thus stabilizing the offset time at this value.

[0234] This application also provides an electronic device, which may include a memory and one or more processors (such as a CPU, GPU, NPU, etc.). The memory and processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the device in the above method embodiments.

[0235] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0236] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0237] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0238] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0239] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0240] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0241] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0243] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0244] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0245] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, including: At the first moment, the synthesis of the first image begins; At the second moment, the synthesis of the second image begins, which is after the first moment. At the third moment, the synthesis of the third image begins. The third moment is after the second moment. The first image, the second image, and the third image are three consecutive frames. The second image is the next frame after the first image, and the third image is the next frame after the second image. The frame rate remains constant from the first time point to the third time point, and the first time interval between the first time point and the second time point is different from the second time interval between the second time point and the third time point.

2. The method according to claim 1, characterized in that, After the synthesis of the first image begins and before the synthesis of the third image begins, the method further includes: Waiting for the TE signal, and in response to the detection of the TE signal, sending the first image to be displayed; Wherein, if the duration of the waiting TE signal corresponding to at least one frame of image is less than the first duration, the second time interval is less than the first time interval; if the duration of the waiting TE signal corresponding to at least one frame of image is greater than the first duration, the second time interval is greater than the first time interval; and the at least one frame of image includes the first image.

3. The method according to claim 1 or 2, characterized in that, When the load on the electronic device is higher than the preset load, the second time interval is shorter than the first time interval.

4. The method according to claim 2, characterized in that, After displaying the first image in response to detecting a TE signal, the method further includes: The offset time is adjusted according to the duration of the waiting TE signal corresponding to the at least one frame of image; The third moment is obtained by subtracting the adjusted offset time from the expected display time of the third image.

5. The method according to claim 4, characterized in that, The step of adjusting the offset time based on the duration of the waiting TE signal corresponding to the at least one frame of image includes: If the duration of the waiting TE signal corresponding to at least one frame of image is less than the first duration, the offset time is increased; if the duration of the waiting TE signal corresponding to at least one frame of image is greater than the second duration, the offset time is decreased.

6. The method according to claim 4 or 5, characterized in that, The at least one frame image is a series of consecutive frames, and the duration of waiting for the TE signal corresponding to the at least one frame image includes the average duration of waiting for the TE signal corresponding to the series of consecutive frames.

7. The method according to any one of claims 4-6, characterized in that, If the duration of the waiting TE signal corresponding to at least one frame of image is greater than the first duration and less than the second duration, the offset time is not adjusted, and the second time interval is the same as the first time interval.

8. The method according to any one of claims 4-7, characterized in that, The step of adjusting the offset time based on the duration of the waiting TE signal corresponding to the at least one frame of image includes: If the load on the electronic device does not exceed the preset load, the offset time is adjusted according to the duration of the waiting TE signal corresponding to the at least one frame of image.

9. The method according to claim 8, characterized in that, The method further includes: If the load on the electronic device exceeds the preset load, the offset time is increased according to the load on the electronic device.

10. The method according to any one of claims 4-9, characterized in that, The electronic device includes a display driver and an image synthesizer; The step of adjusting the offset time based on the duration of the waiting TE signal corresponding to the at least one frame of image includes: The display driver calculates the adjustment amount of the offset time based on the duration of the wait for the TE signal corresponding to the at least one frame of image; The display driver sends the adjustment amount to the image synthesizer; The image synthesizer adjusts the offset time according to the adjustment amount.

11. An electronic device, characterized in that, include: A display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display screen and the one or more memories; the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-10.

13. A computer program product comprising computer instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-10.

Images