Image processing method and electronic device

Abstract

The present application relates to the technical field of electronics, and specifically relates to an image processing method and an electronic device. The method comprises: a first UI thread of a target application drawing a first image frame; the first UI thread triggering a render thread of the target application to render the first image frame; in response to a scrolling operation acting on a target interface of the target application, if the first UI thread experiences stalling while the target interface is being scrolled, a second UI thread of the target application drawing a second image frame, the second UI thread being different from the first UI thread; and the second UI thread triggering the render thread to render the second image frame. The method can reduce frame drops, reduce scrolling lag, and improve user experience.

Description

Image processing methods and electronic devices Technical Field

[0001] This application relates to the field of electronic technology, specifically to an image processing method and an electronic device. Background Technology

[0002] With the widespread use of electronic devices and the rapid development of applications (APPs), users are placing increasingly higher demands on the smoothness of application interfaces. In the process of using applications, swiping, as one of the main ways users interact with the interface, is widely used in scenarios such as news reading, social networks, and e-commerce to enable the swiping of interface content.

[0003] However, in practical applications, it was found that stuttering often occurs during interface scrolling, which seriously affects the user experience. Summary of the Invention

[0004] This application provides an image processing method and an electronic device that can reduce lag during interface scrolling and improve user experience.

[0005] In a first aspect, this application provides an image processing method executed by an electronic device, the method comprising: a first UI thread of a target application drawing a first image frame; the first UI thread triggering a rendering thread of the target application to render the first image frame; in response to a swiping operation on a target interface of the target application, if the first UI thread experiences a lag during the swiping process of the target interface, a second UI thread of the target application drawing a second image frame; the second UI thread being different from the first UI thread; and the second UI thread triggering a rendering thread to render the second image frame.

[0006] The first UI thread can be the main thread, also known as UI thread 1. The second UI thread can be a child thread, also known as UI thread 2.

[0007] The first image frame refers to the image frame drawn before the first UI thread experiences a stutter. The second image frame refers to the image frame drawn by the second UI thread during the stuttering process of the first UI thread; it is also called a supplementary frame. It can be understood that the number of first and second image frames can be one frame or multiple frames.

[0008] First UI thread stuttering refers to the first UI thread running for an extended period. This can be caused by complex business logic corresponding to the image frames being processed, limitations of the electronic device's chip platform, or blocking of the first UI thread due to specific reasons. First UI thread stuttering results in prolonged processing of a single image frame, preventing the processing of subsequent image frames and the triggering of the rendering thread. Consequently, SurfaceFlinger cannot composite the image, causing frame drops and preventing the screen from displaying the image correctly, thus resulting in stuttering.

[0009] The image processing method provided in the first aspect of this application, if the first UI thread experiences lag during the scrolling of the target interface, draws the image through the second UI thread and triggers the rendering thread to execute rendering. This allows SurfaceFlinger to properly composite the image, ensuring the screen displays the image correctly, reducing frame drops and scrolling lag, and improving the user experience. Furthermore, this also reduces the load on the first UI thread and improves its performance.

[0010] In one possible implementation, the target interface includes a swipeable target view, and the second UI thread of the target application draws a second image frame, including: the second UI thread updates the position information of the target view in the second image frame according to the swiping operation and the start time of the lag in the first UI thread.

[0011] The start time of the first UI thread lag refers to the moment when the electronic device detects the lag in the first UI thread and determines that it will cause the interface to lag. In a specific embodiment, the interface lag can be a user-perceptible interface lag.

[0012] The position information of the target view represents the display position of the content corresponding to the target view in the interface. Optionally, the position information of the target view can be updated by updating the position attributes of the target view, such as the offset attribute of the target view.

[0013] In this implementation, the position information of the target view is updated according to the sliding operation and the start time of the lag, so that the relative position of the target view in each second image frame changes, simulating the process of interface sliding, improving the smoothness of sliding, solving the problem of sliding lag, and further improving the user experience.

[0014] In one possible implementation, the second UI thread updates the position information of the target view in the second image frame based on the swipe operation and the start time of the lag in the first UI thread, including: the second UI thread determines a first offset and a first offset direction of the target view relative to the lag start position based on the swipe operation; the lag start position is the position of the target view in the image frame corresponding to the lag start time; the second UI thread updates the position information of the target view based on the first offset and the first offset direction.

[0015] The first offset is also called the sliding offset, and the first offset direction is also called the sliding offset direction.

[0016] It is understandable that the image frame corresponding to the moment when the stuttering begins is also the first image frame, which is the last frame before the stuttering.

[0017] In this implementation, the position information of the target view is updated based on the first and second offsets, thereby causing the entire target view to shift relative to the starting position of the lag. The entire target view refers to the root view and all subviews contained within it. It should be understood that when there are multiple second image frames, the offset of the entire target view relative to the starting position of the lag in each second image frame can gradually increase. This shift of the entire target view relative to the starting position of the lag simulates interface scrolling, improving scrolling smoothness. Furthermore, it is simple and convenient to execute, saves time on the second UI thread, prevents lag in the second UI thread, and further prevents interface lag.

[0018] In one possible implementation, the swipe operation is a hands-off swipe operation. The second UI thread determines the first offset and the first offset direction of the target view relative to the lag start position based on the swipe operation, including: the second UI thread determining the first offset direction based on the swipe curve corresponding to the hands-off swipe operation; the second UI thread determining the lag start position based on the swipe curve; the second UI thread determining the current position of the target view based on the swipe curve; and determining the difference between the current position and the lag start position to obtain the first offset.

[0019] In this implementation, during the off-hand swipe, the target view offset can be accurately predicted based on the swipe curve, simulating a more natural and smooth off-hand swipe and improving the user experience.

[0020] In one possible implementation, the method further includes: if the first UI thread finishes lag, the second UI thread restores the target view to the position where the lag began.

[0021] In one possible implementation, restoring the target view to the stuttering position by the second UI thread includes setting the first offset of the target view to 0.

[0022] In this implementation, after the first UI thread finishes its lag, the target view is restored to the position where the lag began. This allows the first UI thread to accurately fill the newly drawn content into the view container of the target view after it takes over drawing again, improving the accuracy of the interface display and thus enhancing the user experience.

[0023] In one possible implementation, the first UI thread experiences a stutter when the time it takes for the first UI thread to draw an image frame exceeds a preset duration.

[0024] The preset duration can be set to a fixed value according to actual needs, or it can be combined with at least one of the following settings: vertical synchronization signal (Vsync) period, number of currently unconsumed image frames in the frame buffer, and preset maximum allowed number of dropped frames.

[0025] In one possible implementation, the preset duration is x times the Vsync cycle, where x is greater than 1.

[0026] It's understandable that if the first UI thread takes longer than one Vsync cycle to draw a single image frame, it might cause frame drops. Therefore, the preset duration can be set to a value greater than one Vsync cycle. Note that x can be an integer or a decimal.

[0027] In one possible implementation, the preset duration is m times the Vsync period, where m is the number of currently unconsumed image frames in the frame buffer.

[0028] When all the image frames buffered in the frame buffer are consumed and no more can be sent for display, it will cause the interface to stutter. Therefore, the preset duration can be set to m times the Vsync cycle, which can more accurately prevent interface stuttering.

[0029] In one possible implementation, the preset duration is n+m times the Vsync cycle, where n is a preset value, n is an integer, and m is the number of currently unconsumed image frames in the frame buffer.

[0030] The preset value is the maximum allowed number of dropped frames in the specific implementation method. When the number of dropped frames is less than or equal to n, the stuttering caused by the dropped frames is considered imperceptible to the user; when the number of dropped frames is greater than n, the stuttering caused by the dropped frames is considered perceptible to the user. Therefore, setting the preset duration to n+m times the Vsync cycle can prevent stuttering that is perceptible to the user, improve the user experience, and save power consumption.

[0031] In one possible implementation, after the second UI thread of the target application draws the second image frame, the method further includes: if the first UI thread stops lagging, then the first UI thread draws the third image frame; the first UI thread triggers the rendering thread to render the third image frame; the first UI thread draws the fourth image frame; the first UI thread triggers the rendering thread to render the fourth image frame.

[0032] The third image frame is the first image frame drawn by the first UI thread after the lag ends. The fourth image frame is the second image frame drawn by the first UI thread after the lag ends.

[0033] In other words, once the first UI thread finishes rendering, the second UI thread stops drawing image frames, and the first UI thread takes over the rendering of subsequent frames, completing the rendering of each image frame one by one, and triggering the rendering thread to render these image frames frame by frame. In this way, the first UI thread and the second UI thread can achieve a seamless connection in frame rendering, making the interface display more fluid and natural, and improving the user's visual experience.

[0034] In one possible implementation, the method further includes: if the target interface does not stop scrolling when the first UI thread finishes pausing, the second UI thread sends the first parameter of the third image frame to the rendering thread; the first parameter includes the position parameter of the first region in the third image frame and the motion effect parameter corresponding to the first region, the first region being the region in the third image frame with newly added display content compared to the last frame, the second image frame; the rendering thread renders the third image frame based on the first parameter.

[0035] In one possible implementation, the method further includes: if the target interface does not stop scrolling, the second UI thread sends the first parameter of the fourth image frame to the rendering thread; the second parameter includes the position parameter of the first region in the third image frame and the motion effect parameter corresponding to the second region, the second region being the region in the fourth image frame with newly added display content compared to the last second image frame; the rendering thread renders the fourth image frame based on the second parameter.

[0036] The first and second regions are also called motion effect regions.

[0037] Optionally, after the lag ends, if the target interface does not stop scrolling, the second UI thread can send the position parameters of the animation area of ​​the multiple image frames and the corresponding animation parameters of the animation area to the rendering thread in sequence, following the above process. The rendering thread then renders each image frame based on the second parameter.

[0038] It's understandable that the first UI thread moves the target view frame by frame to simulate a sliding effect. However, as the target view moves, the underlying content that was originally obscured by it gradually becomes visible on the screen. Thus, when the lag ends and the first UI thread displays the newly drawn content, the first area of ​​the screen suddenly changes from background content to the newly drawn content, causing abrupt changes in the interface and impacting the user experience.

[0039] In this implementation, after the first UI thread finishes pausing, motion effect parameters are set for the motion effect areas in subsequent multi-frame image sequences. Thus, the continuous multi-frame image sequences after setting the motion effect parameters are displayed on the interface, presenting a motion effect, creating a visual buffer, preventing interface jumps, and improving the user's visual experience.

[0040] In one possible implementation, the motion effect parameters include at least one of color and transparency.

[0041] In one possible implementation, the transparency of the third image frame is greater than that of the fourth image frame.

[0042] In one possible implementation, after the stutter ends and before the swipe ends, the transparency of the motion effect area in the image frame rendered frame by frame by the first UI thread gradually increases.

[0043] Increased transparency means the interface becomes more transparent. This allows the content in the animation area to be gradually revealed clearly, creating a visual buffer, preventing abrupt interface transitions, and improving the user's visual experience.

[0044] In one possible implementation, the color in the animation parameters is the same as the color of the background layer beneath the target view in the target interface. This further enhances the animation effect, allowing newly drawn content in the animation area to be gradually and clearly displayed, creating a better visual buffer and improving the user's visual experience.

[0045] In one possible implementation, the method further includes: determining whether the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame; the fifth image frame is the image frame being drawn when the first UI thread experiences a stutter; if the first UI thread has not completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread triggers the rendering thread to execute the rendering of the fifth image frame; if the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread intercepts the rendering of the fifth image frame.

[0046] Intercepting the rendering of the fifth image frame, for example: not sending the drawing instructions for the fifth image frame to the rendering thread.

[0047] It should be understood that the first UI thread has already completed the calculation of the view position before the second UI thread starts drawing the second image frame (i.e., before frame interpolation begins). Therefore, after the first UI thread of the target application finishes its pause, it updates the properties of the subviews in the target view based on the calculated positions. However, in reality, during the pause, these subviews in the target view have already shifted along with the entire target view. In other words, after the pause ends, the first UI thread adjusts the positions of the subviews in the target view back to their pre-pause positions. As a result, from the display perspective, these subviews will first move along the scrolling direction and then suddenly revert, providing a poor user experience.

[0048] In this implementation, if the first UI thread completes the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread intercepts the rendering of the fifth image frame. In this way, the first UI thread changes the view position to the position before the lag, thereby preventing the interface from falling back and improving the user experience.

[0049] In one possible implementation, determining whether the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame includes: the second UI thread obtaining the target frame number from the first UI thread when the first UI thread starts to lag; the first UI thread sending the target frame number to the second UI thread, the target frame number being the frame number currently held by the first UI thread; if the first UI thread has already started executing the animation() function for the fifth image frame, then the target frame number is the number of the fifth image frame; if the first UI thread has not yet started executing the animation() function for the fifth image frame, then the target frame number is the frame number of the image frame preceding the fifth image frame; the second UI thread using the target frame number as the frame number of the second image frame; if the frame number of the second image frame is different from that of the fourth image frame, then it is determined that the first UI thread has not completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame; if the frame number of the second image frame is the same as that of the fourth image frame, then it is determined that the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame.

[0050] In this implementation, by using the target frame number, it is possible to easily, accurately, and quickly determine whether the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, thereby improving the algorithm's running efficiency.

[0051] In one possible implementation, after the first UI thread triggers the target application's rendering thread to render the first image frame, the method further includes: using SurfaceFlinger to composite the first image frame; and displaying the first image frame on the screen.

[0052] After the second UI thread triggers the rendering thread to render the second image frame, the method also includes: using SurfaceFlinger to composite the second image frame; and displaying the second image frame on the screen.

[0053] After the first UI thread triggers the rendering thread to render the third image frame, the method also includes: using SurfaceFlinger to composite the third image frame; and displaying the third image frame on the screen.

[0054] After the first UI thread triggers the rendering thread to render the fourth image frame, the method also includes: using SurfaceFlinger to synthesize the fourth image frame; and displaying the fourth image frame on the screen.

[0055] It should be understood that in the embodiments of this application, the names of the data are not distinguished for different processing stages of the same image frame. However, this does not mean that the processing results or the generated data content of each processing stage are the same. For example, drawing the first image frame, rendering the first image frame, compositing the first image frame, and displaying the first image frame represent the drawing, rendering, compositing, and display stages of generating the first image frame, but do not mean that the data used and generated in drawing, rendering, compositing, and display are the same. In fact, the data generated in each processing stage, as well as the form of the data, are different, but these data are all specific to the first image and therefore are not specifically distinguished.

[0056] In one possible implementation, the target interface includes a swipeable target view. The swiping of the target interface is a hands-free swipe. During the swiping of the target interface, if the first UI thread lags, the second UI thread of the target application draws a second image frame. This includes: when it is determined that the target view begins to swipe away from the hand, determining whether the first UI thread has started to lag; if the first UI thread starts to lag, the second UI thread draws at least one second image frame; determining whether the lag of the first UI thread has ended; if the lag of the first UI thread has ended, the second UI thread stops drawing the second image frame.

[0057] In this implementation, the detection of whether the first UI thread is lagging only begins after the target interface starts to swipe away from the user. This reduces the number of times the first UI thread is detected for lag, streamlines the algorithm process, and saves device power consumption.

[0058] In one possible implementation, determining whether the first UI thread has started to lag includes: monitoring a first request, which is a request sent by the first UI thread to SurfaceFlinger to request the Vsync-app signal; if no first request is detected within n consecutive Vsync cycles starting from the first moment, then the number m of unconsumed image frames in the frame buffer at the current moment is obtained; n is a preset value, and both n and m are positive integers, and the first moment is the moment when the first request was most recently detected; if no first request is detected within m consecutive Vsync cycles starting from the second moment, then the first UI thread is determined to be laggy; the second moment is the moment n Vsync cycles after the first moment.

[0059] In one possible implementation, the method further includes: taking the n+m Vsync cycles after the first moment as the start moment of the lag in the first UI thread.

[0060] This implementation can accurately and quickly identify user-perceptible lag, reduce the number of times the second UI thread is triggered for drawing, save algorithm steps, and save device power consumption.

[0061] One possible implementation involves determining whether the first UI thread has stopped freezing, including: if the first request is detected after the freeze begins, then the freeze of the first UI thread is determined to have ended.

[0062] The first request is also the Vsync-app request.

[0063] In one possible implementation, the method further includes: the first UI thread sending a first notification to the second UI thread each time the rendering thread is triggered to render an image frame; determining whether the first UI thread's lag has ended includes: if the second UI thread receives the first notification after the lag begins, then determining that the first UI thread's lag has ended.

[0064] The first notification, also known as the drawing notification, is used to notify the first UI thread that a rendering has been triggered.

[0065] In one possible implementation, determining whether the first UI thread's lag has ended includes: after the lag begins, obtaining a target count, where the target count is the number of image frames drawn by the first UI thread among the unconsumed image frames in the frame buffer; if the target count is greater than 0, then determining that the first UI thread's lag has ended.

[0066] In one possible implementation, if the first UI thread lags during the sliding of the target interface, the second UI thread of the target application draws a second image frame, including: if the target application is determined to be an application in the application whitelist and the target interface includes a target view, the second UI thread of the target application draws a second image frame during the sliding of the target interface if the first UI thread lags; the target view is a view in the view whitelist corresponding to the target application.

[0067] In this implementation, the second UI thread is triggered to draw the second image frame, i.e., the sliding optimization function is triggered, only when the target application is an application in the application whitelist and the target interface includes a view in the view whitelist corresponding to the target application. This can save algorithm steps and reduce the power consumption of electronic devices.

[0068] In one possible implementation, the method further includes: when the target application's interface jumps, determining whether the target interface after the jump includes a target view; if the target interface includes a target view, determining whether the target application's process has completed initialization; initialization includes creating a second UI thread for the target application in the current process; if the target application's process has not completed initialization, performing initialization.

[0069] In this implementation, initialization is performed before the target application's process has finished initializing, in order to create a second UI thread for the target application in the current process. This ensures the operation of the scrolling optimization function, reduces screen scrolling lag, and improves the user experience.

[0070] In one possible implementation, when the target application's interface changes, it is determined whether the target interface after the change includes the target view. This includes determining whether the target interface includes the target view when the target application's thread performs window layout or view layout.

[0071] When the interface switches, the window layout and / or view layout will change. Therefore, the recognition of the target view is triggered by the window layout or view layout, which simplifies the algorithm and improves the efficiency of the algorithm.

[0072] In one possible implementation, the target view includes at least one of ScrollView, ListView, or RecyclerView.

[0073] Secondly, this application provides an apparatus included in an electronic device, which has the function of implementing the behaviors of the electronic device in the first aspect and possible implementations thereof. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-described functions. For example, a receiving module or unit, a processing module or unit, etc.

[0074] Thirdly, this application provides an electronic device, which includes a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to execute any one of the methods in the first aspect of the technical solution.

[0075] Fourthly, this application provides a chip system including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof.

[0076] Optionally, the chip system may also include memory, which is connected to the processor via circuitry or wires.

[0077] Alternatively, the chip system may also include a communication interface.

[0078] Fifthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any one of the methods in the first aspect of the technical solution.

[0079] Sixthly, this application provides a computer program product, which includes computer program code that, when executed on an electronic device, causes the electronic device to perform any one of the methods in the first aspect of the technical solution. Attached Figure Description

[0080] Figure 1 is a schematic diagram illustrating the cause analysis of sliding jamming according to an embodiment of this application;

[0081] Figure 2 is a schematic diagram of an example of a sliding lag interface provided in an embodiment of this application;

[0082] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0083] Figure 4 is a software structure block diagram of an example electronic device provided in an embodiment of this application;

[0084] Figure 5 is a flowchart illustrating an example of an image processing method provided in an embodiment of this application;

[0085] Figure 6 is a schematic diagram of interface changes during a sliding process provided in an embodiment of this application;

[0086] Figure 7 is a schematic diagram illustrating the principle of an image processing method provided in an embodiment of this application;

[0087] Figure 8 is a flowchart illustrating an example of a configuration parsing process provided in an embodiment of this application;

[0088] Figure 9 is a schematic flowchart of an example view recognition process provided in an embodiment of this application;

[0089] Figure 10 is a schematic diagram illustrating the principle of view recognition provided in an embodiment of this application;

[0090] Figure 11 is a schematic flowchart of another image processing method provided in an embodiment of this application;

[0091] Figure 12 is a schematic diagram of a stuttering detection principle provided in an embodiment of this application;

[0092] Figure 13 is a flowchart illustrating an example of a stuttering detection process provided in an embodiment of this application;

[0093] Figure 14 is a schematic diagram of a stuttering detection principle provided in an embodiment of this application;

[0094] Figure 15 is a flowchart illustrating another example of a stuttering detection process provided in an embodiment of this application;

[0095] Figure 16 is a flowchart illustrating an example of a parallel frame interpolation process provided in an embodiment of this application;

[0096] Figure 17 is a schematic diagram of an example of a motion effect area provided in an embodiment of this application;

[0097] Figure 18 is a schematic diagram of an example of interface changes provided in an embodiment of this application;

[0098] Figure 19 is a schematic diagram of an example of image processing provided in an embodiment of this application;

[0099] Figure 20 is another schematic diagram of interface changes provided in an embodiment of this application;

[0100] Figure 21 is a flowchart illustrating another example of an image processing method provided in an embodiment of this application;

[0101] Figure 22 is a schematic diagram of an example interface provided in an embodiment of this application;

[0102] Figure 23 is another example of an interface diagram provided in an embodiment of this application. Detailed Implementation

[0103] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0104] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0105] References to "one embodiment" or "some embodiments" as described in this application specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0106] For ease of understanding, the examples provided are for reference only and are related to the concepts in the embodiments of this application.

[0107] 1. Image frame

[0108] An image frame, also simply called a frame or frame data, refers to a single, still image, the smallest unit in a user interface display. A frame can be understood as a still image; rapidly displaying multiple consecutive frames creates the illusion of motion. Frame rate refers to the number of frames refreshed per second, or the number of times the graphics processor in an electronic device refreshes the screen per second. A higher frame rate results in smoother and more realistic animation. The more frames per second, the smoother the displayed motion.

[0109] It should be noted that before an image frame is displayed on the screen, it usually needs to go through processes such as drawing, rendering, compositing, and sending to the display.

[0110] 2. Frame drawing

[0111] Frame rendering, also known as drawing frames or drawing image frames, refers to the rendering of images on a display interface. A display interface can consist of one or more views, each drawn by a view system. Each view consists of a root view and subviews. A subview corresponds to a small component within a view; for example, one subview might correspond to a symbol within the interface.

[0112] Frame rendering is implemented by the user interface (UI) thread. Frame rendering mainly includes handling user interaction logic and updating the view. Updating the view includes, but is not limited to, measuring and laying out the view, generating rendering instructions, and calculating animation positions. Generating rendering instructions refers to converting the measurement, layout, and rendering operations of the view tree into a series of rendering instructions. These rendering instructions can form a display list. In other words, rendering instructions describe the layout and style of the view.

[0113] 3. Frame rendering

[0114] Frame rendering involves coloring or adding 3D effects to a drawn view. For example, 3D effects can include lighting effects, shadow effects, and texture effects.

[0115] 4. Frame Composition

[0116] Frame compositing is the process of combining layer data from one or more rendered views into an image frame. The composited image frames can then be cached in a frame buffer.

[0117] 5. Send Display

[0118] Sending the composited image frame to the display screen for display.

[0119] 6. The vertical synchronization signal used in the application (Vsync-app)

[0120] To coordinate the application's rendering, compositing, and display processes with the screen refresh rate and resolve screen tearing issues, the application's vertical synchronization signal (Vsync-app) can trigger the rendering, compositing, and display of application image frames. Vsync-app can be generated by combining a surface compositing (SurfaceFlinger) with a Vsync calibration model. The period of the Vsync-app signal is called the Vsync period, which is related to the refresh rate of the electronic device.

[0121] It should be understood that in the embodiments of this application, the names of the data are not distinguished for different processing stages of the same image frame. However, this does not mean that the processing results or the generated data content of each processing stage are the same. For example, drawing the first image frame, rendering the first image frame, compositing the first image frame, and displaying the first image frame represent the drawing, rendering, compositing, and display stages of generating the first image frame, but do not mean that the data used and generated in drawing, rendering, compositing, and display are the same. In fact, the data generated in each processing stage, as well as the form of the data, are different, but these data are all specific to the first image and therefore are not specifically distinguished.

[0122] The technical problems faced by this application are explained below.

[0123] Swiping is one of the primary ways users interact with the interface, and the smoothness of swiping is closely related to user experience. However, in practical applications, it has been found that stuttering can easily occur during interface swiping, affecting the user experience. The specific reasons are analyzed as follows:

[0124] During screen scrolling, the application's UI thread needs to draw image frames one by one. Then, the UI thread triggers the render thread to render each frame. However, in some cases, after a user performs a scrolling operation, due to the complexity of the business logic corresponding to the scrolling operation, and / or the limited capabilities of the electronic device's chip platform, the application's UI thread may need to run for a long time, or may be blocked. In this embodiment, the prolonged running and blocking of the UI thread are collectively referred to as UI thread stuttering. In systems such as Android, the UI thread is the only thread responsible for user interaction and frame drawing; all processing in the UI thread is performed serially. This means that when the UI thread stutters, the responsiveness of the entire application is affected. Specifically, UI thread stuttering prevents the processing of subsequent business logic, the drawing of subsequent image frames, and the triggering of the render thread to render the current and subsequent image frames, resulting in frame drops and consequently, screen scrolling stuttering.

[0125] For example, Figure 1 is a schematic diagram illustrating the cause analysis of scrolling lag provided in an embodiment of this application. As shown in Figure 1, when the Vsync-app1 signal arrives, the UI thread of a certain application draws frame F1, generates the drawing instruction for frame F1, and sends the drawing instruction to the Render thread, triggering the Render thread to render frame F1. When the Vsync-app2 signal arrives, SurfaceFlinger synthesizes frame F1. Subsequently, when the Vsync-app3 signal arrives, the screen displays frame F1.

[0126] Suppose that at time T1, after the UI thread has finished processing the data for frame F1, a swipe operation from the user is received. The UI thread responds to this swipe operation by drawing frame F2. However, due to the complexity of the business logic triggered by this swipe operation, the UI thread experiences lag and a long runtime (shown as black fill in Figure 1). As a result, from Vsync-app2 to Vsync-app5, the UI thread cannot trigger the Render thread to execute rendering. The Render thread misses rendering three image frames, causing SurfaceFlinger to miss compositing these three image frames, thus resulting in the continuous display of frame F1 on the screen.

[0127] After the UI thread freezes, the Render thread is triggered to render frame F2. Then, the UI thread draws an image frame (let's say frame F6) that matches the finger's swipe trajectory or the swipe curve after the finger leaves the screen. After the UI thread freezes again, the Render thread is triggered to render frame F2. Then, when Vsync-app6 arrives, SurfaceFlinger composites frame F2, and when Vsync-app7 arrives, frame F2 is displayed on the screen. Simultaneously, when Vsync-app6 arrives, the UI thread draws frame F6 and triggers the Render thread to render frame F6. When Vsync-app7 arrives, SurfaceFlinger composites frame F6, and when Vsync-app8 arrives, frame F6 is displayed on the screen.

[0128] In other words, UI thread lag causes the screen to freeze for at least 3 Vsync cycles (lag duration t) at frame F1 (shown as a diagonal pattern fill in the image), after which the screen jumps directly from frame F2 to frame F6. Tests have shown that some applications experience lag durations t of over 90 milliseconds (ms), and the sudden jump from frame F2 to frame F6 is noticeably jarring for users, negatively impacting the user experience.

[0129] Figure 2 shows a schematic diagram of a sliding lag interface provided in an embodiment of this application. Corresponding to the schematic diagram shown in Figure 1, after frame F1 is synthesized and displayed, the screen of the electronic device displays interface 201 as shown in Figure 2(a). After frame F2 is synthesized and displayed, the screen of the electronic device displays interface 202 as shown in Figure 2(b). After frame F6 is synthesized and displayed, the screen of the electronic device displays interface 203 as shown in Figure 2(c). The actual response process is as follows: the user performs an upward sliding operation on interface 201 shown in Figure 2(a). The electronic device does not respond within time t, that is, the screen continuously displays interface 201 within time t. Then, the content in the interface moves upward slightly, displaying interface 202 as shown in Figure 2(b), and then suddenly jumps to interface 203 as shown in Figure 2(c). It is evident that a noticeable sliding lag occurs on the interface.

[0130] In related technologies, the following methods are used to reduce the frame drop rate during scrolling, thereby solving the problem of scrolling stuttering:

[0131] 1) Reduce business logic

[0132] Some implementations reduce the complexity of the application's business logic, thereby shortening the duration of the UI thread's continuous operation and reducing frame drops during scrolling. This approach fundamentally solves the scrolling stuttering problem and improves scrolling smoothness by reducing unnecessary computation and processing. However, this method has limited applicability. On one hand, for some applications, the business logic cannot be reduced, otherwise it may affect the normal operation of the business. On the other hand, reducing the application's business logic requires modifying the application's business logic code, which is not possible for the electronic device running the application; therefore, this solution cannot be implemented on the electronic device side.

[0133] 2) Increase the frequency of the central processing unit (CPU).

[0134] In some implementations, increasing the CPU frequency can temporarily boost the processing power of electronic devices, thereby reducing lag and improving the scrolling experience. However, increasing the CPU frequency leads to increased CPU power consumption, which in turn affects the battery life of electronic devices. Furthermore, there is a maximum limit to CPU frequency; excessively increasing the frequency can cause the device to overheat or become unstable. For example, some phones frequently increase their CPU frequency under high load. While this may temporarily improve the performance of the electronic device and solve the lag problem, after prolonged use, the phone overheats significantly, battery life decreases noticeably, and the system becomes unstable.

[0135] 3) Adjust refresh rate

[0136] In some implementations, reducing the screen refresh rate of electronic devices can decrease the number of dropped frames and improve scrolling smoothness. However, lowering the refresh rate may negatively impact the user experience, especially in high refresh rate scenarios such as games or fast-scrolling interfaces. A low refresh rate can cause visual stuttering, operational lag, and other issues, still affecting the user experience.

[0137] 4) Adjust the scroll curve.

[0138] A sliding curve refers to the dynamic response relationship between time and sliding displacement, or time and interface sliding speed, during a sliding operation. In some embodiments, adjusting the sliding curve smooths changes in interface sliding speed to appropriately reduce frame drops and optimize sliding smoothness and response speed. However, this approach may lead to negative optimization. Specifically, adjusting the sliding curve essentially requires considering both user experience and sliding frame drop data. A good user experience for interface sliding speed and a good response speed for good sliding frame drop data may not be consistent. Therefore, while adjusting the sliding curve to reduce interface sliding speed can decrease the number of frame drops, it may make the user perceive the interface sliding speed as too slow, negatively impacting the overall user experience.

[0139] In conclusion, the methods in the relevant technologies cannot effectively solve the problem of interface scrolling lag.

[0140] In view of this, embodiments of this application provide an image processing method in which, during a period of lag in the application's UI thread 1 (main thread, also known as the first UI thread), image frames are drawn through a sub-thread (also known as UI thread 2 or the second UI thread) and the Render thread is triggered to perform rendering. In this way, during the lag in UI thread 1, the Render thread can submit the layer data of each rendered frame to SurfaceFlinger, thereby enabling the display screen to show each image frame, reducing frame drops, reducing scrolling lag, and improving the user experience.

[0141] For ease of explanation, in the following embodiments, the function of reducing scrolling stutter and improving scrolling smoothness using the method provided in this application will be referred to as the scrolling optimization function. Of course, this name is only an example and does not constitute any limitation on this application.

[0142] The structure of the electronic device to which the image processing method provided in the embodiments of this application is applied will be described below.

[0143] The image processing method provided in this application can be applied to electronic devices that can install applications (APPs), such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0144] For example, Figure 3 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0145] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0146] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0147] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0148] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0149] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0150] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0151] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0152] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0153] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0154] Figure 4 is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0155] As shown in Figure 4, the application package can include applications such as camera, gallery, calendar, call, map, navigation, browser, social media, music, video, and games. Optionally, the applications in the application layer can be system applications or third-party applications; there is no limitation on this.

[0156] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0157] Specifically, the application framework layer may include a Java framework layer (referred to as the framework layer) and a native framework layer (referred to as the native layer, also known as the C++ framework layer). As shown in Figure 4, in this embodiment, the framework layer may include a configuration parsing module, a view system, an activity manager service (AMS), and a window manager service (WMS), etc.

[0158] The configuration parsing module is used to read and parse the application's configuration file or a pre-configured configuration file in the system. In this application, the configuration parsing module is used to obtain the status of the feature switch for the sliding optimization function, and when the feature switch is enabled, it parses the configuration file for the sliding optimization function to obtain information such as the application whitelist and view whitelist for the sliding optimization function, as well as the maximum allowed number of dropped frames. After the configuration parsing module completes the parsing, it can send the parsing result to the view system. In some embodiments, the configuration parsing module can be a module of an application such as a mobile phone manager, and can run in the mobile phone manager process.

[0159] The view system manages and presents the various views of the user interface. It provides a set of classes and interfaces that enable developers to build rich and diverse display interfaces. A display interface can consist of one or more views. For example, a display interface including a text message notification icon can include views that display text and views that display images.

[0160] As shown in Figure 4, in this embodiment, the view system may include a drawing module, a view recognition module, a parallel frame interpolation module, and a stuttering detection module. The drawing module is used to draw image frame data. The drawing module achieves drawing by creating a UI thread 1 for each application. The view recognition module is used to identify whether the target interface includes a target view when the sliding optimization feature switch is enabled and the target application's interface jumps to the target interface. The target application belongs to the application whitelist. The target view refers to the view in the view whitelist corresponding to the target application. The stuttering detection module is used to identify the start and end of stuttering in the target application's UI thread 1, based on the maximum allowed number of dropped frames and the number of currently unconsumed frames (also called unconsumed buffers) in the frame buffer, when the target interface includes the target view. Optionally, the stuttering detection module can achieve stuttering detection of the target application's UI thread 1 by creating a stuttering detection thread for the target application. When the target application's UI thread 1 starts to stutter, the stuttering detection module notifies the parallel frame interpolation module to start frame interpolation. When the lag in the target application's UI thread 1 ends, the lag detection module notifies the parallel frame interpolation module to stop interpolation. The parallel frame interpolation module creates a second UI thread for the target application. Upon receiving the interpolation notification from the lag detection module, it uses UI thread 2 to update the properties of the target view and sends drawing instructions to the target application's Render thread, triggering the Render thread to render the interpolated frames. Additionally, the parallel frame interpolation module can also send the position and animation parameters of the animation areas for multiple image frames to the Render thread even when the lag in UI thread 1 ends and the scrolling continues.

[0161] WMS is used to manage window applications, and is responsible for managing window views, such as the position, size, and layout of application windows. Additionally, WMS can manage window focus changes.

[0162] AMS is responsible for the unified scheduling and management of activities in all processes within the system. Through activity management, AMS can control application startup, shutdown, and activity transitions. Application startup can include warm startup, cold startup, and gentle startup.

[0163] Of course, the framework layer may also include content providers, phone managers, resource managers, and notification managers, etc., which are not shown in Figure 4.

[0164] The native layer includes a surface compositing (SurfaceFlinger) and a rendering module. The rendering module is used for execution-based rendering. Specifically, the rendering module can create a corresponding Render thread for each application to render image frames for each application. In this application, the Render thread of the target application executes rendering after receiving drawing instructions from UI thread 1 or UI thread 2 of the target application. Furthermore, after receiving the position parameters and animation parameters of the animation area sent by UI thread 2 of the target application, and the drawing instructions sent by UI thread 1 of the target application, the Render thread of the target application renders the view animation based on the position parameters and animation parameters of the animation area, and the drawing instructions.

[0165] SurfaceFlinger is used to retrieve buffers from the frame buffer and synthesize them into image frames. In this embodiment, SurfaceFlinger is also used to provide the stuttering detection module with the number of currently unconsumed buffers.

[0166] Content providers are used to store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.

[0167] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0168] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0169] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0170] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0171] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0172] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0173] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0174] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0175] A 2D graphics engine is a graphics engine for 2D drawing.

[0176] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0177] For ease of understanding, the following embodiments of this application will take an electronic device with the structure shown in Figures 3 and 4 as an example, and in conjunction with the accompanying drawings and application scenarios, will specifically illustrate the image processing method provided by the embodiments of this application.

[0178] First, taking application 1 as an example, the image processing method provided in this application embodiment will be described from the perspective of process. Optionally, application 1 can be any application with scrollable content in its interface, such as a social application, browser, video application, etc. For example, Figure 5 is a flowchart illustrating an example of image processing provided in this application embodiment, and Figure 6 is a schematic diagram illustrating an example of image processing provided in this application embodiment. Referring to Figures 5 and 6 together, the method includes the following steps, and the executing entity of the following steps is an electronic device.

[0179] S101. In response to the user's operation, start application 1 and create UI thread 1 and Render thread of application 1.

[0180] Specifically, in response to the user's launch of application 1, the electronic device creates a process for application 1 and various threads related to that process, including the UI thread and Render thread. The UI thread is responsible for handling user interaction logic and view updates. The UI thread is also known as the main thread.

[0181] In this embodiment, in addition to the main thread, another thread is involved, which has a similar function to the main thread and is also responsible for updating the view of the interface. For ease of distinction, in this application, the main thread is referred to as UI thread 1 (also known as the first UI thread), and the sub-thread responsible for updating the interface view is referred to as UI thread 2 (also known as the second UI thread). UI thread 2 and UI thread 1 run in parallel and do not affect each other.

[0182] Specifically, UI thread 1 is responsible for drawing image frames one by one, that is, processing user interaction logic and view updates frame by frame, and triggering the Render thread to execute the rendering of each frame. UI thread 2 is responsible for supplementing the view updates of some frames when UI thread 1 is lagging, and triggering the Render thread to execute the rendering of these supplementary frames. These supplementary frames are referred to as supplementary frames (or interpolated frames) in the following embodiments of this application.

[0183] S102. The data of frame F1 (also known as the first image frame) is processed by UI thread 1 of application 1. The view of frame F1 includes scrollable view 1 (also known as the target view).

[0184] The data processing for frame F1 includes: processing business logic related to frame F1, laying out the view of frame F1, and generating drawing instructions for frame F1.

[0185] Frame F1 is the image frame to be displayed in the interface after application 1 starts, and it is also the image frame to be displayed in the interface before UI thread 1 lags. For example, Figure 6 is a schematic diagram of interface changes during a sliding process provided in an embodiment of this application. As shown in Figure 6(a), frame F1 is, for example, the image frame corresponding to interface 601. The view of frame F1 includes a scrollable view 1. View 1 corresponds to the scrollable area 602 in interface 601.

[0186] Specifically, UI thread 1 of application 1 responds to the launch operation, processes the preset launch logic, and constructs a view tree, laying out all the views in the view tree. This view tree includes a scrollable view 1, such as a ScrollView, ListView, or RecyclerView.

[0187] S103, UI thread 1 of application 1 triggers Render thread of application 1 to execute rendering of frame F1.

[0188] Specifically, UI thread 1 of application 1 sends a drawing instruction for frame 1 to the Render thread of application 1. In response to the drawing instruction for frame F1, the Render thread of application 1 performs a rendering operation on frame F1 and obtains the layer data of frame F1.

[0189] S104. Based on the rendering result of frame F1, display interface 1, which includes a slidable area corresponding to view 1.

[0190] Specifically, the Render thread of application 1 submits the layer data of frame F1 to SurfaceFlinger. SurfaceFlinger combines the layer data of frame F1 to obtain frame F1. SurfaceFlinger sends frame F1 to the display to obtain interface 1. Interface 1 is, for example, interface 601 shown in Figure 6(a).

[0191] It's understandable that UI thread 1 doesn't need to draw frames if the user doesn't perform any operations on interface 1 and the content of interface 1 doesn't need to be changed. If the user performs an operation on interface 1, and / or the content of interface 1 needs to be updated, UI thread 1 starts drawing the next frame. The following example illustrates this using a swipe operation on interface 1.

[0192] S105. In response to the user performing an upward swipe operation in the swipeable area of ​​interface 1, UI thread 1 of application 1 processes the data of frame F2.

[0193] Processing the data of frame F2, i.e. drawing frame F2, includes: responding to the user's swipe operation, processing business logic, updating the view properties of frame 2, and generating drawing instructions for frame F2.

[0194] Frame F2 is the image to be displayed on the interface in response to a swipe operation. In this embodiment, frame F2 is also the image frame being drawn when UI thread 1 experiences a lag. Frame F2 is also called the fifth image frame.

[0195] Specifically, UI thread 1 of application 1 responds to the user's swipe operation and handles the related business logic. Simultaneously, UI thread 1 of application 1 updates the view. The update for view 1 is as follows: the root view (i.e., the container view) of view 1 remains unchanged; the positions of the subviews within view 1 are updated to move the content originally displayed within the swipe area upwards; and new subviews added to the layout of view 1 are added to supplement the swipe area with new content.

[0196] S106. If UI thread 1 of application 1 lags during the interface scrolling process, the data of the supplementary frame Fn (also known as the second image frame) is processed by UI thread 2 of application 1.

[0197] Processing the data in the supplementary frame Fn includes updating the position attribute of view 1 in each supplementary frame Fn according to the sliding curve, so that view 1 moves (moves upward in this embodiment). It should be understood that moving view 1 means that the root view, subviews, etc. of view 1 move together.

[0198] The supplementary frame Fn refers to the image frame to be displayed in the interface that is generated during the UI thread 1 of application 1 when it is stuck.

[0199] Optionally, the number of supplementary frames Fn can be one frame or multiple frames. In one specific embodiment, supplementary frames can be generated based on the application's vertical synchronization (Vsync-app) signal.

[0200] Optionally, UI thread 2 of application 1 can be created when the process of application 1 is created, or it can be created after the process of application 1 is created, when it is determined that preset conditions are met. The preset conditions are, for example, that the process of application 1 is running in the foreground, and that the views in the interface include preset scrollable views.

[0201] Optionally, UI thread 2 can run on the small cores of the electronic device.

[0202] Optionally, there can be multiple criteria for determining whether UI thread 1 is lagging. In one specific embodiment, UI thread 1 is considered lagging when its processing time for a certain image frame exceeds a preset duration. The preset duration can be set to a fixed value according to actual needs, or it can be determined based on the Vsync cycle. For example, the preset duration can be x times the Vsync cycle, where x is a value greater than 1. Another example is m times the Vsync cycle, where m is the number of unconsumed image frames in the frame buffer (also simply called the number of unconsumed frames), and m is an integer. Yet another example is n+m times the Vsync cycle, where n is the maximum allowed number of dropped frames, and n is an integer.

[0203] S107. The UI thread 2 of application 1 triggers the Render process of application 1 to execute the rendering of the supplementary frame Fn.

[0204] S108, Rendering result display interface n based on supplementary frame Fn.

[0205] S109. After the UI thread 1 of application 1 stops lagging, restore view 1 to its original position through UI thread 2 of application 1, and process the data of frame Fx through UI thread 1 of application 1.

[0206] Frame Fx represents the image frame to be displayed on the interface after the lag ends. Frame Fx may include, for example, the third image frame, the fourth image frame, and so on. Within frame Fx, the sub-views of View 1 are laid out and filled by UI thread 1 according to business logic requirements. It can be understood that Fx varies depending on the duration of the lag. For example, with a lag duration of 4 Vysnc cycles, Fx would be frame F6.

[0207] After the UI thread 1 freezes, UI thread 2 can restore view 1 to its original position (the position before the freeze, or the position where the freeze started), that is, set the sliding offset of view 1 to 0. In this way, view 1, including its root view (i.e., the container view), is restored to its original position, and UI thread 1 updates the properties of the child views based on the root view and populates new child views, so that the interface is displayed correctly.

[0208] It should be noted that after the UI thread 1 of application 1 finishes its lag, it may or may not trigger the Render thread to render frame F2, depending on whether UI thread 1 completed the position calculation of view 1 at the start of the lag. For more details, please refer to the following embodiments. In this embodiment, we will illustrate the example of UI thread 1 not triggering the Render thread to render frame F2.

[0209] S110. The UI thread 1 of application 1 triggers the Render thread of application 1 to execute the rendering of frame Fx.

[0210] S111, Display interface x based on the rendering result of frame Fx.

[0211] It is understandable that as the upward movement of view 1 in each supplementary frame Fn increases sequentially, the content of the sliding area in the interface corresponding to the supplementary frame gradually moves upward, presenting the effect of interface sliding.

[0212] Referring again to Figure 6, after the user performs an upward swipe operation within the sliding area 602 of interface 601, UI thread 1 of application 1 experiences a pause. During this pause, UI thread 2 of application 1 renders supplementary frames Fn by executing the above process. Figures (b) to (d) in Figure 6 exemplarily show schematic diagrams of several interfaces n corresponding to supplementary frames Fn. As shown, by setting the position attribute of view 1 corresponding to the sliding area 602, view 1 as a whole (including the root view and subviews of view 1) gradually moves upward, thereby causing the content of the sliding area 602 to gradually move upward, presenting the effect of interface sliding.

[0213] After the lag ends, UI thread 2 of application 1 restores view 1 to its original position, and UI thread 1 of application 1 continues to process frame Fx and triggers the Render thread to render frame Fx. The interface x corresponding to frame Fx can be interface 603 as shown in Figure (e) of Figure 6.

[0214] In other words, during the lag of UI thread 1, UI thread 2 processes the data of the supplementary frame and triggers the rendering of the supplementary frame. The supplementary frame simulates the interface sliding, connecting the interface before the lag of UI thread 1 (interface 601 shown in Figure 6(a)) and the interface after the lag (interface 603 shown in Figure 6(a)). This makes the interface sliding smoother and more natural, reduces the lag, and improves the user experience.

[0215] It should be understood that Figures (b) to (d) in Figure 6 are only examples. In practice, the number of supplementary frames Fn can be more, and the position movement of view 1 each time can be smaller, so that the presented sliding effect is smoother and more natural.

[0216] For example, Figure 7 is a schematic diagram of the principle of an image processing method provided in an embodiment of this application. As shown in Figure 7, a stutter occurs during the processing of frame F2 data by UI thread 1 (shown as black fill in the figure). At this time, the process switches to UI thread 2 to draw supplementary frames Fn frame by frame, and UI thread 2 triggers the Render thread to execute the rendering of each supplementary frame Fn (shown as grid fill in the figure) frame by frame. Then, SurfaceFlinger synthesizes the supplementary frames frame by frame, and the screen displays the supplementary frames frame by frame. When UI thread 1 finishes the stutter, UI thread 1 continues to draw frame Fx. Then, UI thread 1 triggers Render to execute the rendering of frame Fx. Then, SurfaceFlinger synthesizes frame Fx, and the screen displays frame Fx.

[0217] Comparing Figures 1 and 7, it can be seen that the method provided in this embodiment can alleviate the pressure on UI thread 1 during UI thread 1 lag by using parallel UI thread 2. Furthermore, during UI thread 1 lag, UI thread 2 can trigger the Render thread to render supplementary frames, which in turn triggers SurfaceFlinger to synthesize supplementary frames and display them on the screen. These supplementary frames simulate interface scrolling, thus solving the problem of interface scrolling lag, improving the smoothness of interface scrolling, and enhancing the user experience. In addition, when UI thread 2 draws supplementary frames, it only needs to update the position attributes of the scrollable view, resulting in a simple execution process. UI thread 2 is less prone to lag, further ensuring the smoothness of the interface display corresponding to the supplementary frames and further preventing interface scrolling lag.

[0218] The image processing method provided in this application embodiment will be described below from the perspective of functional modules, with reference to Figure 4.

[0219] Optionally, to streamline the algorithm and conserve power consumption of electronic devices, the method provided in this application embodiment can pre-configure an application whitelist and a view whitelist for the sliding optimization function. Applications in the application whitelist are called whitelisted applications, and views in the view whitelist are called whitelisted views. Each whitelisted application corresponds to a view whitelist. When the currently displayed interface of the electronic device is the interface of a whitelisted application, and this interface includes views from the corresponding view whitelist of the whitelisted application, sliding optimization is performed using the method provided in this application embodiment. In other words, the sliding optimization function is triggered when the interface includes a whitelisted view of a whitelisted application. Furthermore, a feature switch can be set for the sliding optimization function, and the state of the feature switch can be configured to control the activation and deactivation of the sliding optimization function. The application whitelist, view whitelist, and feature switch state of the sliding optimization function can be pre-configured in a configuration file.

[0220] For clarity, the method of this application will be described below according to the processes of configuration parsing, view recognition, stuttering detection, and parallel frame interpolation.

[0221] 1. Configuration Analysis

[0222] For example, Figure 8 is a flowchart illustrating a configuration parsing process provided in an embodiment of this application. The method includes:

[0223] S201. The configuration parsing module obtains the configuration file for the sliding optimization function every time the electronic device is powered on.

[0224] Specifically, each time the electronic device is powered on, the configuration parsing module retrieves the configuration file for the sliding optimization function from a preset storage location according to a preset procedure. The configuration file for the sliding optimization function may include, but is not limited to: the on / off status of the sliding optimization function feature, the maximum number of allowed frame drops, the application whitelist, and the view whitelist corresponding to each application in the application whitelist.

[0225] The maximum allowed number of dropped frames is used for subsequent stuttering detection. When the number of dropped frames is less than or equal to the maximum allowed number, the stuttering caused by the dropped frames is considered insignificant and imperceptible to the user, and therefore can be ignored. When the number of dropped frames exceeds the maximum allowed number, the stuttering caused by the dropped frames is considered significant, and the sliding optimization function needs to be activated. The specific value of the maximum allowed number of dropped frames can be set according to actual needs; for example, the number of dropped frames can be 5 or 2.

[0226] The application whitelist includes information about one or more applications. Optionally, the application information can be the package name. Each application in each application whitelist corresponds to a view whitelist. The view whitelist includes information about one or more views corresponding to the application. Optionally, the view information can be the view name. For example, the application whitelist includes application A, application B, and application C. Application A corresponds to a view whitelist a, which includes one or more views of application A. Application B corresponds to a view whitelist b, which includes one or more views of application B. Application C corresponds to a view whitelist c, which includes one or more views of application C.

[0227] Optionally, the views in the whitelist are scrollable views, such as ScrollView, ListView, RecyclerView, etc.

[0228] In one specific embodiment, the configuration file format can be as follows:

[0229] Here, `withoutAvailableBufCount` represents the maximum allowed number of dropped frames. `package_name` represents the package name in the application whitelist. `view_name` represents the view name in the view whitelist corresponding to a given package name.

[0230] After obtaining the configuration file, the configuration parsing module first retrieves the status of the feature switches from the configuration file. Based on the status of the feature switches, the configuration parsing module decides whether to further parse the configuration file.

[0231] In some embodiments, the state of the feature switch can be changed after the electronic device is powered on. Optionally, the state of the feature switch can be changed by the user through a human-computer interaction interface, or by the developer through a background program; there is no limitation on this.

[0232] S202. The configuration parsing module determines whether the state of the feature switch in the configuration file is enabled; if not, proceed to step S203; if yes, proceed to steps S204 to S206.

[0233] S203. The configuration parsing module sets the preset characteristic attribute value to 0 and does not parse the configuration file.

[0234] S204. The configuration parsing module sets the preset characteristic attribute value to 1 and parses the configuration file to obtain the application whitelist, view whitelist and maximum allowed frame loss.

[0235] The default attribute value is, for example, the `prop` value. A `prop` value of 1 indicates that the sliding optimization feature is enabled, and a `prop` value of 0 indicates that the sliding optimization feature is disabled. Of course, 1 and 0 are just examples; other values ​​can also be used to represent enabling and disabling the sliding optimization feature.

[0236] S205. The configuration parsing module sends the application whitelist and view whitelist to the view recognition module in the view system.

[0237] S206. The configuration parsing module sends the maximum allowed number of frame drops to the stuttering detection module in the view system.

[0238] 2. View recognition

[0239] For example, Figure 9 is a schematic flowchart of a view recognition process provided in an embodiment of this application. The method includes:

[0240] S301. When the characteristic attribute value is 1, the view recognition module monitors the interface jumps of each application in the application whitelist.

[0241] S302. When the view recognition module detects a screen jump of an application (referred to as the target application) in the application whitelist, it determines whether the screen after the jump (referred to as the target screen) includes the target view. The target view refers to the view in the view whitelist corresponding to the target application. If yes, then proceed to step S303. If no, then continue to proceed to step S302 to continue recognizing the screen jump of the target application.

[0242] The target application is the application currently running in the foreground that is in the application whitelist. For example, the target application is application 1 mentioned above. The target view is, for example, view 1 mentioned above.

[0243] It is understood that in the embodiments of this application, interface transition of the target application refers to a change in the view of the interface. Specifically, interface transition can include the interface appearing from nothing, for example, the target application going from not being launched to displaying the interface after launch. Application launch includes cold start, warm start, and soft start. Interface transition can also include transitions between interfaces of different attributes or levels, for example, transitioning from the target application's "main interface" to the "discovery interface". Interface transition can also include changes to some views within the same interface, for example, the rewriting or switching of a certain view in the target application's "main interface".

[0244] In other words, when the swipe optimization feature is enabled, if a target application in the whitelist navigates to a different screen, the view recognition module is triggered to perform view recognition on the target application's screen. During the screen navigation, either the window layout or the view layout is triggered. Therefore, as a possible implementation, view recognition can be triggered through window layout and view layout. This allows for accurate and rapid triggering of view recognition, improving the reliability of the swipe optimization feature.

[0245] For example, Figure 10 is a schematic diagram of the principle of view recognition provided in an embodiment of this application. As shown in Figure 10, the managers in the system include AMS, WMS, etc. When application 1 starts, AMS creates the activity of application 1 through the desktop application (launchActivity) and initializes the window. During the process of managing the window through ViewRootImpl, WMS handles the window focus switching through windowFocusChanged() and responds to changes in window size caused by screen rotation or split screen through handleResized(). The window focus switching or window size change triggers the view layout module of the view system to re-layout the view. The view layout module can re-layout the view through functions such as handleVisibilityChange() and set the control state through functions such as setPressed(). In addition, when the application (e.g., a third-party application) view is overridden, the view layout module also re-layouts the view.

[0246] Based on this, the identification of the target view can be triggered by system stubs or application stubs related to window layout. System stubs refer to callback points or triggering mechanisms in the system framework used to respond to critical events. Application stubs refer to callback points or triggering mechanisms in the application's logic used to respond to critical events. In a specific embodiment, the view identification module can be triggered to identify the target view using system window layout stubs such as `windowFocusChanged()` and `handleResized()`, and application view stubs such as `viewOverride()`.

[0247] Optionally, the view recognition module can start from the root view of the target interface and use a hierarchical traversal algorithm to scan the entire view tree of the target interface layer by layer to determine whether the target interface contains the target view.

[0248] S303. The view recognition module determines whether the process of the target application has completed functional initialization based on the initialization flag of the target application. If not, steps S304 to S305 are executed to perform functional initialization. If yes, step S309 is executed.

[0249] Whether the target application's process has completed functional initialization can be understood as whether the target application has completed the initialization related to the sliding optimization function within its current process. Initialization related to the sliding optimization function includes, but is not limited to: creating the target application within its current process. Each time an electronic device cold-starts the target application, a new process for the target application is created, which can then be based on the target application's UI thread 2 and stuttering detection thread. This can be understood as creating the necessary threads within the process, including UI thread 2 and the stuttering detection thread. Furthermore, if the target application is killed, its process is destroyed, and all threads are also destroyed, including UI thread 2 and the stuttering detection thread. Therefore, upon the next launch of the target application, the sliding optimization function needs to be initialized again, and UI thread 2 and the stuttering detection thread need to be recreated to enable the services related to the sliding optimization function.

[0250] Based on this, assuming the target interface contains the target view, we can first determine whether the target application has created UI thread 2 and a stuttering detection thread in the current process. If not, we execute steps S304 to S309 to perform function initialization and create UI thread 2 and the stuttering detection thread of the target application in the current process. If they have been created, we execute step S309 to perform frame interpolation and stuttering detection processes based on UI thread 2 and the stuttering detection thread of the target application.

[0251] To facilitate querying and management, the initial identification module can set corresponding initialization flags for each whitelisted application to indicate the initialization status of each application's functions. Specifically, taking the target application as an example, the target application's initialization flag can be set to uninitialized (e.g., UNINIT) by default. After the target application's process is created, if the target view of the target application is identified for the first time, the target application's initialization flag is changed to initialized (e.g., INITED). After the target application's process is destroyed, all threads of the application are also destroyed, and the initialization flag is restored to UNINIT.

[0252] In this embodiment, after identifying that the target view is included in the target interface of the target application, it is determined whether the initialization flag of the target application is uninitialized, which essentially means determining whether this is the first time the target view has been identified since the target application created its current process. If the initialization flag of the target application is uninitialized (e.g., UNINIT), it means that this is the first time the target view has been identified since the target application created its current process, and steps S304 to S309 are executed to perform function initialization; if the initialization flag of the target application is initialized (e.g., INITED), it means that this is not the first time the target view has been identified since the target application created its current process, and step S309 is executed.

[0253] S304, The view recognition module notifies the lag detection module to perform function initialization.

[0254] S305. In response to the notification from the view recognition module, the stuttering detection module creates a stuttering detection thread for the target application in the current process (hereinafter referred to as the stuttering detection thread of the target application) to start the stuttering detection service for the target application.

[0255] After the stuttering detection thread for the target application is created, the stuttering detection module can provide stuttering detection services to the target application based on this thread. In other words, the stuttering detection module can detect whether the UI thread 1 of the target application stutters during scrolling of the target interface, based on the target application's stuttering detection thread. When the stuttering detection module detects the UI thread 1 of the target application, it sends a frame interpolation notification to the parallel frame interpolation module.

[0256] S306, The view recognition module notifies the parallel frame interpolation module to perform function initialization.

[0257] S307. In response to the notification from the view recognition module, the parallel frame interpolation module creates UI thread 2 of the target application in the current process (hereinafter referred to as UI thread 2 of the target application) to start the parallel frame interpolation service for the target application.

[0258] The target application's UI thread 2 runs in parallel with the target application's UI thread 1. After the target application's UI thread 2 is created, the parallel frame interpolation module can provide parallel frame interpolation services to the target application based on the target application's UI thread 2 when it receives a frame interpolation notification sent by the target application's stuttering detection thread.

[0259] S308, The view recognition module modifies the initialization flag of the target application to be initialized (e.g., INITED).

[0260] S309. The view recognition module updates the value of the recognition result variable of the target application to the target view.

[0261] Specifically, the view recognition module can maintain a recognition result variable for each whitelisted application. This variable records the view recognition result. The initial value of the recognition result variable can be NULL (or empty, 0, etc.), indicating that the target interface does not contain a whitelisted view. When a whitelisted view is recognized in a whitelisted application, the value of the recognition result variable for that whitelisted application is updated to reflect the recognized whitelisted view.

[0262] Taking the target application as an example, the initial value of the target application's recognition result variable is NULL. If the target application's target interface is found to include a target view (e.g., view 1), then the target application's recognition result variable will be updated to view 1.

[0263] The identification result variable can be a global variable, and the parallel frame interpolation module, stuttering detection module, etc. can obtain the value of the identification result variable.

[0264] 3. Lag detection

[0265] Optionally, in the image processing method provided in this embodiment, the stuttering detection is mainly used to detect stuttering during the interface swiping process after the user lifts their hand. It should be understood that when the user performs a swipe operation, the target view moves with the user's swipe. When the user's swipe operation ends, that is, after the user lifts their hand (also called releasing the hand), if the swipe speed at the time of lifting the hand is not zero, the electronic device determines the swiping curve based on the speed at the time of lifting the hand, and draws and renders the interface based on the swiping curve to present the effect that the interface continues to swipe for a period of time after the hand is lifted, gradually decelerating until it stops. For ease of description, in the following embodiments, the continued swiping of the interface after the user lifts their hand is referred to as off-hand swiping or inertial swiping. That is to say, the stuttering detection in this embodiment can be applied during the user's off-hand swiping.

[0266] Of course, in other embodiments, stuttering detection may be performed both during the swipe operation and during the off-hand swipe, and this is not limited.

[0267] For example, Figure 11 is a schematic flowchart of another image processing method provided in an embodiment of this application, which includes the following steps S401 to S407. The execution entity of each of the following steps is a stuttering detection module. The stuttering detection module can implement the following process based on the stuttering detection thread of the target application.

[0268] S401. When the lag detection module determines that the value of the recognition result variable of the target application is the target view, it identifies the off-hand swipe of the target view and determines whether the target view has started to swipe off-hand. If yes, then proceed to step S402; if no, then continue to proceed to step S401.

[0269] S402. The lag detection module detects whether the UI thread 1 of the target application has started to lag; if the UI thread 1 of the target application has not started to lag, then proceed to step S403; if the UI thread 1 of the target application starts to lag, then proceed to step S404.

[0270] Specifically, after executing step S305 and creating a lag detection thread for the target application in the current process, the lag detection module can determine whether lag detection is needed for the current interface based on the value of the identification result variable. When the value of the identification result variable is NULL, it indicates that the target interface does not include the target view, and the sliding optimization function does not need to be activated. Therefore, lag detection is not required for UI thread 1 of the target application. When the value of the identification result variable is the target view, it indicates that the target interface includes the target view, and the sliding optimization function needs to be activated. Therefore, lag detection is required for UI thread 1 of the target application.

[0271] If the value of the identification result variable for the target application is the target view, the lag detection module can continuously monitor the target view for off-hand swipes. Each time an off-hand swipe is detected, the lag detection process is initiated to check if lag occurs in UI thread 1 of the target application. The lag detection process ends when the off-hand swipe is detected to have ended.

[0272] As one possible implementation, the lag detection module can identify the start of a hands-free swipe as follows: The lag detection module identifies the UP event in the user's swipe operation on the target view and determines the swipe speed corresponding to the UP event. If the lag detection module identifies an UP event in the swipe operation on the target view, and the swipe speed corresponding to the UP event is greater than 0, then it determines that the target view has started a hands-free swipe.

[0273] As another possible implementation, the stuttering detection module can also obtain the sliding curve from the rendering module when it detects an UP event in a user's swipe operation on the target view. It can be understood that the rendering module can determine the sliding curve based on the swipe speed when the UP event occurs, as well as the position of the view in the interface. If the swipe speed when the UP event occurs is equal to 0, it indicates that no release swipe is needed, and therefore no sliding curve is generated. Therefore, if the stuttering detection module does not obtain a sliding curve from the rendering module, it means that the release swipe has not started.

[0274] In other words, during the swipe away from the target view, a stuttering detection process is performed on the UI thread 1 of the target application. If stuttering is detected in the UI thread 1 of the target application, steps S404 and S501 to S505 in subsequent embodiments are executed to perform frame interpolation.

[0275] The specific implementation process of step S402 can be found in the embodiment shown in Figure 13 below.

[0276] S403. The lag detection module determines whether the target view has ended its swipe away from the hand; if yes, it returns to step S401; if no, it returns to step S402.

[0277] S404, the stuttering detection module sends a frame interpolation notification and the stuttering start time to the parallel frame interpolation module.

[0278] The frame interpolation notification is used to notify the start of frame interpolation.

[0279] Step S405 is executed after step S404.

[0280] S405. The lag detection module determines whether the lag in UI thread 1 of the target application has ended; if yes, proceed to step S407; if no, proceed to step S406.

[0281] The specific implementation process of step S405 can be found in the embodiment shown in Figure 15 below.

[0282] S406. The stuttering detection module determines whether the off-hand swipe of the target view has ended; if yes, proceed to step S407; if no, return to step S402.

[0283] Optionally, the stuttering detection module can determine whether the hands-free sliding has ended based on the sliding curve. The sliding curve characterizes the response relationship between time and sliding speed, or time and displacement (or sliding distance). Based on the sliding curve, the total duration of the hands-free sliding can be determined. Therefore, when the stuttering detection module detects the start of the hands-free sliding, it can start a timer for the total duration of the hands-free sliding; if the timer ends, it indicates that the hands-free sliding has ended. Of course, this method is only one example; other methods can also be used to determine whether the hands-free sliding has ended, and this application does not limit this to any particular method.

[0284] S407, The stuttering detection module sends a frame interpolation end notification to the parallel frame interpolation module.

[0285] The frame interpolation end notification is used to notify the user to stop frame interpolation.

[0286] After step S407 is completed, you can return to step S401 and repeat the process from step S401 to S407 to repeatedly check whether the swipe is off-hand and whether the UI thread 1 of the target application is lagging.

[0287] In this embodiment, a stuttering detection module detects the start and end of stuttering in the target application's UI thread 1, triggering a parallel frame interpolation module to interpolate frames during the UI thread 1 stuttering period. This prevents frame interpolation when UI thread 1 is not stuttering, thus preventing interference with the normal operation of UI thread 1 and ensuring normal interface display. Furthermore, the method in this application addresses stuttering during interface scrolling. Therefore, the stuttering detection module detects the start and end of off-hand scrolling in the target view, performing stuttering detection on the target application's UI thread 1 only during off-hand scrolling. This saves algorithm steps and device power consumption. Additionally, it makes it easier to predict view offset during off-hand scrolling, resulting in a more natural and smooth simulated interface scrolling effect and improved user experience.

[0288] The specific implementation process of step S402 will be further explained below.

[0289] Understandably, to reduce resource consumption, when an application needs to start drawing and rendering a new image frame, the application's UI thread can send a Vsync-app request (also known as the first request) to SurfaceFlinger. After receiving the request, SurfaceFlinger generates a Vsync-app signal based on the Vsync calibration model and sends it to the application's UI thread.

[0290] Based on this, this application determines whether UI thread 1 experiences lag by monitoring the Vsync-app requests during the target view's swipe gesture. Furthermore, in this embodiment, when detecting lag, lag with a frame drop count less than or equal to the maximum allowed frame drop count has a minimal impact on user experience and is considered acceptable; therefore, no frame interpolation or swipe optimization is performed in this case. For lag with a frame drop count greater than the maximum allowed frame drop count, frame interpolation and swipe optimization are performed. This simplifies the algorithm flow, saves power, and prevents misidentification of lag and excessive frame interpolation.

[0291] For example, Figure 12 is a schematic diagram of a stuttering detection principle provided in an embodiment of this application. As shown in Figure 12, the stuttering detection module may include a Vsync request monitoring unit, a swipe recognition unit, an available frame detection unit, and a stuttering recognition unit. The Vsync request monitoring unit is used to monitor whether the UI thread 1 of the target application sends a Vsync-app request to SurfaceFlinger. The swipe recognition unit is used to monitor the off-hand swipe of the target view, including the start and end of the off-hand swipe of the target view. That is, steps S401 and S406 above can be implemented by the swipe recognition unit. The available frame detection unit is used to obtain the number of unconsumed frames in the frame buffer from SurfaceFlinger. The stuttering recognition unit is used to identify the start and end of the stuttering of the UI thread 1 of the target application based on the monitoring results of the Vsync request monitoring unit, the recognition results of the swipe recognition unit, and the number of unconsumed frames obtained by the available frame detection unit. That is, the stuttering recognition unit can implement steps S402 and S405 above.

[0292] For example, Figure 13 is a flowchart illustrating a lag detection process provided in an embodiment of this application. As shown in Figure 13, step S402, where the lag detection module detects whether the UI thread 1 of the target application has started to lag, includes:

[0293] S4021, the Vsync request monitoring unit monitors the Vsync-app request sent from the UI thread 1 of the target application to SurfaceFlinger when it is determined that the value of the identification result variable of the target application is the target view.

[0294] As one possible implementation, the Vsync request monitoring unit can subscribe to Vsync-app request events from the target application. When the target application's UI thread 1 sends a Vsync-app request to SurfaceFlinger, it notifies the Vsync request monitoring unit.

[0295] As another possible implementation, the Vsync request monitoring unit can also actively monitor whether the target application's UI thread 1 sends a Vsync-app request to SurfaceFlinger.

[0296] As another possible implementation, the Vsync request monitoring unit subscribes to the Vsync-app request event of the target application to SurfaceFlinger. When SurfaceFlinger receives a Vsync-app request sent by the UI thread 1 of the target application, it notifies the Vsync request monitoring unit.

[0297] S4022, when the Vsync request monitoring unit detects that the UI thread 1 of the target application sends a Vsync-app request to SurfaceFlinger, it sends a notification 1 to the stuttering identification unit.

[0298] Notification 1 is used to notify that a target application has been detected sending a Vsync-app request to SurfaceFlinger.

[0299] It is understandable that steps S4021 and S4022 can be continuously executed.

[0300] S4023. Each time the stuttering identification unit receives notification 1, it triggers the execution of the stuttering identification algorithm once.

[0301] Optionally, the swipe recognition unit can notify the stuttering recognition unit after recognizing the start of a swipe away from the target view. Upon receiving the notification that a swipe has started away from the target view, the stuttering recognition unit executes the stuttering recognition algorithm once after each notification (as described in notification 1).

[0302] The stuttering detection algorithm includes the following process:

[0303] a. The stuttering identification unit determines whether, starting from time t1 (also known as the first time), it has not received notification 1 for n Vsync cycles; where time t1 is the time when the UI thread 1 of the target application last sent a Vsync-app request to SurfaceFlinger, n is the maximum allowed number of dropped frames, and n is a positive integer; if yes, then steps b and c are executed; if no, then the stuttering identification algorithm is re-executed, that is, step a is continued.

[0304] b. The stuttering detection unit obtains the number of unconsumed frames m from SurfaceFlinger through the available frame detection unit. m is a positive integer.

[0305] c. The stuttering identification unit determines whether, starting from time t2 (also known as the second time), notification 1 has not been received for m Vsync cycles; time t2 is the time when n Vsync cycles have been continuously received since time t1. If yes, then step d is executed; if no, then the stuttering identification algorithm is re-executed, i.e., the execution returns to step a.

[0306] d. The stuttering identification unit determines that the UI thread 1 of the target application starts to stutter, and takes time t3 as the start time of the stuttering; time t3 is the time that lasts for n+m Vsync cycles from time t1.

[0307] In other words, the stuttering detection unit determines whether the target application's UI thread 1 has not sent a Vsync-app request to SurfaceFlinger for any of the n Vsync-app cycles starting from the moment t1 when the last Vsync-app request was sent to SurfaceFlinger. If so, the stuttering detection unit determines whether the target application's UI thread 1 has not sent a Vsync-app request to SurfaceFlinger for m consecutive Vsync cycles starting from moment t2. If so, the unit determines that the target application's UI thread 1 has started to stutter.

[0308] Specifically, when UI thread 1 needs to process a new image frame, it sends a Vsync-app request to SurfaceFlinger. If, starting from time t1, before the nth Vsync-app cycle, UI thread 1 of the target application sends another Vsync-app request, it means that UI thread 1 of the target application has not experienced any stuttering, or the number of dropped frames caused by stuttering has not reached n frames; in other words, the number of dropped frames has not reached the maximum allowed number of dropped frames. In this case, the number of dropped frames is considered to be within an acceptable range, no frame interpolation is needed, and monitoring for stuttering should continue.

[0309] If UI thread 1 does not send a Vsync-app request again within n consecutive Vsync-app cycles from the moment t1 when it last sent a Vsync-app request to SurfaceFlinger, it indicates that UI thread 1 of the target application is continuously processing a certain image frame without triggering the processing of the next image frame. That is, UI thread 1 is experiencing a stutter, and this stutter will cause the frame buffer to be short of n image frames, i.e., causing n frame drops, which is equal to the maximum allowed number of frame drops. At this time (i.e., moment t2), the stutter caused by UI thread 1 is considered to be within an acceptable range, and no frame supplementation is required.

[0310] It's understandable that SurfaceFlinger retrieves one unconsumed image frame from the frame buffer and synthesizes it each time a Vsync-app signal arrives, before sending it to the display. This process is also known as consuming an image frame. If, from time t2 until all image frames cached in the frame buffer have been consumed, the lag in UI thread 1 is still not resolved, the number of dropped frames will exceed the maximum allowed number of dropped frames. Moreover, the lag caused by dropped frames will be reflected in the interface after all frames cached in the frame buffer have been consumed, meaning the interface will start to lag when scrolling.

[0311] Based on this, if no notification 1 is received for n consecutive Vsync cycles starting from time t1, this application executes steps b and c to determine whether the target application's UI thread 1 has not sent a Vsync-app request to SurfaceFlinger for m consecutive Vsync cycles starting from time t2. That is, it determines whether UI thread 1 requests the Vsync-app signal during the consumption of frames buffered in the frame buffer, thereby determining whether the UI thread 1's stutter has ended. If the target application's UI thread 1 sends a Vsync-app request to SurfaceFlinger for m consecutive Vsync cycles, it indicates that UI thread 1 requested the Vsync-app signal during the consumption of frames buffered in the frame buffer, thus indicating that UI thread 1 has resolved the stutter. Therefore, the number of lost frames remains n frames, which is still within an acceptable range, and no frame replacement is needed.

[0312] If the target application's UI thread 1 does not send a Vsync-app request to SurfaceFlinger for m consecutive Vsync cycles, it means that UI thread 1 did not request the Vsync-app signal during the consumption of frames cached in the frame buffer, indicating that UI thread 1 is still lagging, and frame interpolation is required. Furthermore, frame interpolation should begin at the moment when the frames cached in the frame buffer are completely consumed, i.e., time t3. This allows for immediate frame interpolation just before the interface starts to lag, achieving seamless image frame transitions and further improving scrolling smoothness.

[0313] Optionally, regarding step a, the specific implementation method can be as follows: After the lag detection unit starts swiping away from the screen, if it receives notification 1 sent by the Vsync request monitoring unit, it starts a timer for n Vsync cycles (called timer 1); the lag detection unit determines whether it receives notification 1 sent by the Vsync request monitoring unit again before the timer ends; if it receives notification 1 sent by the Vsync request monitoring unit again before the timer ends, it restarts timer 1 for n Vsync cycles (that is, returns to step a); if it does not receive notification 1 sent by the Vsync request monitoring unit again until the timer ends, it executes step b.

[0314] Optionally, there is no limitation on the timing method. Timing can be done through timers, timers, or by checking timeout information.

[0315] Optionally, regarding step c, the specific implementation can be as follows: The stuttering identification unit starts a timer (called timer 2) with a duration of m Vsync cycles at time t2. The stuttering identification unit determines whether it receives notification 1 from the Vsync request monitoring unit again before the end of timer 2; if it receives notification 1 from the Vsync request monitoring unit again before the end of timer 2, it cancels timer 2 and starts timer 1 with a duration of n Vsync cycles (that is, returns to step a); if it does not receive notification 1 from the Vsync request monitoring unit again until the end of timer 2, it determines that UI thread 1 has started to stutter and executes step d.

[0316] The following explanation is based on the accompanying drawings.

[0317] For example, Figure 14 is a schematic diagram of a stuttering detection principle provided in an embodiment of this application. As shown in Figure 14, at time t0, the swipe recognition unit detects the start of a free swipe and notifies the stuttering recognition unit. At time t1, after time t0, the UI thread 1 of the target application needs to process the data of frame A, and therefore sends a Vsync-app request 1 to SurfaceFlinger. After receiving the Vsync-app signal, UI thread 1 begins to process the data of frame A. At the same time, after the Vsync request monitoring unit detects the Vsync-app request, it sends notification 1 to the stuttering recognition unit.

[0318] After receiving notification 1, the stuttering detection unit starts timer 1, which lasts for n (assuming n is 2) Vsync cycles, and determines whether it receives notification 1 again from the Vsync request monitoring unit before timer 1 ends. Assuming that by the end of timer 1, i.e., time t2 (t2 = t1 + 2 * TVsync, where TVsync is a Vsync cycle), the stuttering detection unit does not receive notification 1 again from the Vsync request monitoring unit, then at time t2, the stuttering detection unit obtains the number of currently unconsumed frames m from SurfaceFlinger. Assuming m is 2, the stuttering detection unit starts timer 2, which lasts for 2 Vsync cycles, and determines whether it receives notification 1 again from the Vsync request monitoring unit before timer 2 ends. Assuming that by the end of timer 2, i.e., time t3 (t3 = t1 + 5 * TVsync), the stuttering detection unit does not receive notification 1 again from the Vsync request monitoring unit, then the stuttering detection unit determines that the UI thread 1 of the target application has started stuttering, and takes time t3 as the start time of this stuttering. The stuttering detection unit notifies the parallel frame interpolation module to start frame interpolation.

[0319] In summary, during the swipe-off process of the target view, the stuttering detection unit executes a stuttering detection algorithm every time it detects a Vsync-app request from the target application's UI thread 1. The stuttering detection algorithm identifies whether the target application's UI thread 1 has not sent a Vsync-app request to SurfaceFlinger for a continuous period of n+m Vsync cycles since the last time the target application sent a Vsync-app request, thus determining whether UI thread 1 is experiencing stuttering. If UI thread 1 has not sent a Vsync-app request to SurfaceFlinger for a continuous period of n+m Vsync cycles, then UI thread 1 is determined to be stuttering; otherwise, UI thread 1 is determined not to be stuttering.

[0320] The method provided in this embodiment can easily, quickly, and accurately identify whether the UI thread 1 of the target application is lagging during the swipe of the target view. In addition, the method allows the target application to have a certain number of dropped frames (maximum allowed number of dropped frames n), which not only reduces the number of times the frame interpolation algorithm is executed, saving device power consumption, but also prevents the slow response of the target application's UI thread 1 from being misidentified as lag, thereby preventing excessive frame interpolation and improving the interface display effect.

[0321] The following provides a further explanation of step S405.

[0322] In this application embodiment, the following methods are provided for detecting the end of a lag:

[0323] Method 1:

[0324] For example, Figure 15 is a flowchart illustrating another instance of a stuttering detection process provided in an embodiment of this application. Please refer to Figures 12 and 15 together. The above step S405, where the stuttering detection module determines whether the stuttering of UI thread 1 of the target application has ended, includes:

[0325] S4051, the stuttering identification unit obtains the target count from the SurfaceFlinger through the available frame detection unit. The target count is the number of image frames drawn by the UI thread 1 of the target application among the unconsumed image frames in the frame buffer.

[0326] Specifically, SurfaceFlinger can set a target count variable corresponding to the target application, and when storing an image frame in the frame buffer, SurfaceFlinger can determine whether the UI thread processing the image frame is UI thread 1 or UI thread 2. If the stored image frame is processed by UI thread 1 of the target application, then count = count + 1. If the stored image frame is processed by UI thread 2 of the target application, then count remains unchanged. In this way, the stuttering detection unit can determine whether the frame buffer stores an image frame processed by UI thread 1 of the target application based on the value of the target count variable.

[0327] S4052. The stuttering identification unit determines whether the target count is greater than 0. If the target count is greater than 0, it determines that the stuttering of UI thread 1 of the target application has ended. If the target count is equal to 0, it returns to step S4051.

[0328] Method 2:

[0329] In some embodiments, the UI thread 1 of the target application can send a drawing notification (also called a first notification) to the stuttering detection unit each time it sends a drawing instruction to the Render thread. The drawing notification informs the UI thread 1 of the target application that a rendering has been triggered. After detecting that the UI thread 1 of the target application has started to stutter, if the stuttering detection unit receives the drawing notification from the UI thread 1, it determines that the stuttering of UI thread 1 has ended. It can be understood that, corresponding to this implementation of identifying the end of stuttering, as another implementation, the aforementioned detection of the start of stuttering can also be achieved by receiving the drawing notification from UI thread 1 instead of monitoring the Vsync-app request of UI thread 1.

[0330] Method 3:

[0331] In other embodiments, after sending a frame interpolation notification to the parallel frame interpolation module, if the stuttering identification unit receives notification 1 from the Vsync request monitoring unit, it determines that the UI thread 1 of the target application has ended its stuttering. In other words, if it detects that the UI thread 1 of the target application requests a Vsync-app signal from SurfaceFlinger, it determines that the stuttering of UI thread 1 has ended.

[0332] In this embodiment, by recognizing the end of the target view swipe and the end of the lag, timely notification is given to end frame interpolation. This prevents conflicts between image frames processed by UI thread 2 and UI thread 1, improving the interface display effect.

[0333] 4. Parallel frame interpolation

[0334] For example, Figure 16 is a flowchart illustrating a parallel frame interpolation process provided in an embodiment of this application. As shown in Figure 16, the parallel frame interpolation stage includes the following process. The execution entity of the following process is the parallel frame interpolation module, which can complete the frame interpolation of the target application based on the UI thread 2 of the target application. Therefore, in the following embodiments, the UI thread 2 of the target application is used as the execution entity for description.

[0335] S501, the target application's UI thread 2 responds to the frame interpolation notification by sending a Vsync-app request to SurfaceFlinger.

[0336] After receiving the Vsync-app request, SurfaceFlinger sends the Vsync-app signal to the UI thread 2 of the target application according to the Vsync cycle.

[0337] S502, the UI thread 2 of the target application determines the sliding offset (also known as the first offset) and the sliding offset direction (also known as the first offset direction) based on the sliding curve, the start time of the lag, and the Vsync cycle each time the Vsync-app signal arrives.

[0338] The sliding offset represents the position of the target view at the current moment, and the amount of displacement relative to the target view position at the start of the lag (also known as the lag start position). The sliding offset direction represents the direction of displacement of the target view position at the current moment relative to the target view position at the start of the lag.

[0339] Specifically, as mentioned above, the sliding curve represents the relationship between time and displacement, or time and sliding speed, during the off-hand sliding period. Therefore, based on the sliding curve, the displacement (or sliding distance) of the target view at each moment during the off-hand sliding period can be determined. In one embodiment, the UI thread 2 of the target application can first determine the displacement d1 of the target view at the start of the lag based on the sliding curve, and then determine the displacement d2 of the target view at the current moment (i.e., x Vsync cycles after the start of the lag, where x is a positive integer) based on the sliding curve. Then, the difference between d2 and d1 is calculated to obtain the sliding offset.

[0340] Optionally, the direction of the sliding offset can be characterized by the sign of the sliding offset amount.

[0341] As one possible implementation, UI thread 2 can immediately execute the drawing of the first supplementary frame after receiving the supplementary frame notification, without waiting for the Vsync-app signal. This allows for faster start-up of the drawing and rendering of the supplementary frame, earlier synthesis of the supplementary frame, and timely display of the supplementary frame, further preventing frame drops and improving the smoothness of the interface display.

[0342] S503, UI thread 2 of the target application updates the properties of the target view in the supplementary frame based on the sliding offset and sliding offset direction.

[0343] Specifically, the UI thread 2 of the target application updates the properties of the target view of the target application, including setting the offset property to the "slide offset" calculated in step S502, updating the offset direction property to the "slide offset direction" determined in step S502, or setting the offset property that includes both offset and slide direction to a slide offset with positive and negative values.

[0344] Optionally, the target application's UI thread 2 updates the target view's properties using the animation() function within the drawFrame() function, generating supplementary frame data.

[0345] S504. The UI thread 2 of the target application sends drawing instructions for the supplementary frames to the Render thread frame by frame, triggering the rendering of the supplementary frames.

[0346] After receiving the drawing instructions for the supplementary frame, the Render thread executes the rendering of the supplementary frame. Since the target application's UI thread 2 only updates the properties of the target view, without changing the layout and content of its subviews, the Render thread can render the target view according to its updated position. This means that if the target view is moved, its root view and all its subviews—that is, all content contained within the target view—will move along with it. After rendering, the Render thread sends the obtained layer data to SurfaceFlinger for compositing. SurfaceFlinger then composites the layer data and stores the resulting image frame in the frame buffer, awaiting display.

[0347] It can be understood that the target application's UI thread 2 calculates the sliding offset frame by frame, updates the target view's properties frame by frame, and triggers the Render thread to render frame by frame. The interface then displays each supplementary frame, thus presenting the gradual movement of the target view, as shown in Figure 6 above. In other words, although the target application's UI thread 1 is lagging and unable to render the next image frame, the target application's UI thread 2 changes the target view's position properties frame by frame during the UI thread 1 lag, triggering the Render thread to render frame by frame. This achieves frame supplementation and the effect of interface sliding, solving the lag problem and improving the user experience.

[0348] The following section provides further explanation of the connection between UI thread 2 and UI thread 1 after the lag ends.

[0349] Referring again to Figure 16, the image processing method provided in this embodiment further includes:

[0350] S505. After receiving the frame interpolation end notification, UI thread 2 of the target application stops frame interpolation and sets the sliding offset of the target view to 0.

[0351] Once the UI thread 1 of the target application finishes lagging, the Render thread will be triggered to render, thereby filling the target view's view container with the drawn content. Therefore, in this step, the sliding offset of the target view is restored to 0, that is, the target view (including its container view) is restored to its original position, so that the new subviews laid out by UI thread 1 can be accurately filled into the target view's container view.

[0352] It's understandable that the above process achieves the effect of interface scrolling by moving the target view. However, as the target view moves, the underlying content that was originally obscured by the target view is revealed and displayed on the interface, causing the interface to change abruptly after the lag ends.

[0353] Specifically, as shown in Figure 6, taking View 1 corresponding to region 602 as the target view, and with a white background on the lower layer of View 1 as an example, as the position attribute of View 1 is updated, the slidable region 602 gradually moves upward, and a white background appears in the lower half of the interface. When the UI thread 2 of the target application finishes its lag and View 1 returns to its original position, the data processed by UI thread 1 fills the container view of View 1. Then, the lower half of the interface will suddenly change from white to newly drawn content, as shown in Figures (d) to (e) of Figure 6. The content of region 604 suddenly changes from white to video content, which makes the user feel a significant interface jump and affects the user experience.

[0354] In view of this, in the method provided in the embodiments of this application, when the UI thread 2 of the target application ends the lag, animation effects are added to the newly drawn content. The animation effects gradually transition the interface change process, reduce the visual impact of the interface change on the user, and improve the user's visual experience.

[0355] Referring again to Figure 16, the image processing method provided in this embodiment further includes:

[0356] S506. After receiving the frame interpolation end notification, if the hand-off swipe of the target view has not ended, the UI thread 2 of the target application determines the position parameters and motion parameters of the motion effect area corresponding to the multi-frame image frame.

[0357] The position parameters and motion parameters of the animation area can also be collectively referred to as the first parameter.

[0358] The motion effect area, also known as the first area or the second area, refers to the area where the motion effect is displayed, that is, the area where new content needs to be displayed compared to the last supplementary frame. Optionally, referring to Figure 17, the position parameters of the motion effect area include the horizontal (x) and vertical (y) coordinates of the motion effect area in a preset coordinate system, as well as the width and height parameters of the motion effect area. Referring to Figure 17, the preset coordinate system is, for example, the xoy coordinate system with the top left vertex of the electronic device as the coordinate system. Then, (x, y) is the coordinate of the top left vertex of the motion effect area 1701, the width represents the length of the side of the motion effect area 1701 along the x-axis, and the height represents the length of the side of the motion effect area 1701 along the y-axis. Optionally, the position parameters of the motion effect area can be represented as (x, y, width, height). The sliding area in a multi-frame image can gradually increase in size along the sliding direction.

[0359] Animation parameters are used to define the desired effect of the animation. Optionally, animation parameters can include one or more of the following: color, transparency, etc. For example, the color in the animation parameters can be the background color of the target view. If the background color is white, the animation effect can be a white transparency gradient, with multiple frames corresponding to white colors and the transparency increasing, for example, gradually changing from a small value (such as 10%) to 100%, to present the process of the interface gradually becoming transparent from pure white, finally revealing the newly drawn content.

[0360] In one specific embodiment, the number of frames for implementing the animation effect, as well as the values ​​of the animation effect parameters, can be set based on the time difference between the current moment and the end moment of the hand swipe, so that the animation effect is displayed before the swipe ends. For example, if the time difference between the current moment and the end moment of the hand swipe is sufficient to display 4 frames, then the number of frames for implementing the animation effect can be 4. Taking the swipe operation as an upward swipe with a white background of 0 transparency as an example, the number of frames for implementing the animation effect can be, for example, 4 frames. Then, the position parameters and animation effect parameters of the animation effect area for the 4 image frames are set. Specifically, in the position parameters of the animation effect area for the 4 image frames, the x and width remain unchanged, the y decreases sequentially, and the height increases sequentially. The transparency of the 4 image frames increases sequentially, for example, it can be 25%, 50%, 75%, and 100% respectively.

[0361] In another implementation, the sliding animation parameters can also be fixed values. In this case, the transparency of the image frame closest to the end of the swipe can be set to its maximum value to ensure the interface displays correctly.

[0362] S507. When the target application's UI thread 2 arrives each time the Vsync-app signal arrives, it sends the position parameters and motion parameters of the motion effect area corresponding to one frame of image to the Render thread.

[0363] S508, the Render thread renders each image frame one by one based on the animation area, animation parameters and the drawing instructions of the UI thread 1 of the target application.

[0364] For example, Figure 18 shows a schematic diagram of the interface changes of an electronic device after setting the animation effect. As shown in Figure 18, the UI thread 2 of the target application sends the animation area and animation parameters to the Render thread frame by frame. As the UI thread 1 of the target application updates the view frame by frame and triggers the Render thread to render, the image frames displayed by the electronic device in sequence are shown in Figures (a) to (e) of Figure 18. It can be seen that the transparency of each image frame gradually increases, so that the newly drawn content is gradually displayed, reducing visual impact and bringing a better visual experience to the user.

[0365] Additionally, if the UI thread 1 lag ends just as the hand-off swipe also ends, no animation is set, meaning steps S506 to S508 are not executed.

[0366] If UI thread 1 does not end the lag, but the swipe ends when the user releases their hand, then no animation effect will be set, meaning steps S506 to S508 will not be executed.

[0367] Furthermore, the inventors discovered that in some scenarios, UI thread 1 might have already calculated the position (referred to as the target position) of the subviews in the target view that need to be repositioned before UI thread 2 begins frame interpolation. Therefore, after the UI thread 1 of the target application experiences a lag, it updates the attributes of the subviews in the target view based on the calculated target position. In reality, during the lag, these subviews in the target view have already shifted along with the entire target view; see the above-mentioned example regarding parallel frame interpolation for details. That is, after the lag ends, UI thread 1 adjusts the positions of the subviews in the target view back to their pre-lag positions. As a result, from the display perspective, these subviews will first move along the sliding direction and then suddenly revert, providing a poor user experience. This will be further explained below with reference to the schematic diagram in Figure 19 and the interface diagram in Figure 20.

[0368] As shown in Figure 19, when UI thread 1 processes data for frame B, it first executes the `doFrame()` function upon receiving the `Vsync-app` signal. During the execution of `doFrame()`, functions such as `animation()` are called. UI thread 1 calculates the target position during `animation()`. However, if UI thread 1 experiences a stutter, because a stutter detection process is required before triggering UI thread 2 to perform frame interpolation, UI thread 2's frame interpolation may occur after `animation()`. In other words, UI thread 1 has already completed the target position calculation before frame interpolation.

[0369] For example, the image frame before the lag is shown in Figure 20(a). After UI thread 2 adds several supplementary frames, the target view has gradually moved up, and the interface displayed in the last frame before the lag ends is shown in Figure 20(b). Figures 20(a) and (b) are the same as Figures 6(a) and (d) above, and will not be described again.

[0370] After the lag ends, if the Render thread renders the content in the target view according to the target position calculated by UI thread 1 during animation(), the interface corresponding to the first frame after the lag ends is shown in Figure 20(c). Comparing Figures 20(a), (b), and (c), it can be seen that the interface first gradually slides upwards to the interface shown in Figure 20(b), and then suddenly slides downwards to the interface shown in Figure 20(c), that is, the interface exhibits a regression phenomenon, which affects the user experience.

[0371] In view of this, in the method provided in the embodiments of this application, UI thread 2 intercepts the rendering of frames that have completed the target position calculation before frame interpolation, so as to prevent the interface from reverting and improve the user's visual experience.

[0372] Referring to Figure 21, after step S501 and before S504, the method provided in this embodiment further includes the following process:

[0373] S601. After receiving the frame interpolation notification, UI thread 2 of the target application requests the frame number from UI thread 1 of the target application.

[0374] S602, in response to the request from UI thread 2, the target application's UI thread 1 sends the currently held frame number (also known as the target frame number) to the target application's UI thread 2.

[0375] Specifically, the target application's UI thread 1 generates the frame number of the current frame at the start of each frame's doFrame(), or animation(). Optionally, the frame number can be generated sequentially, that is, by incrementing the frame number of the previous frame by 1.

[0376] When UI thread 2 of the target application obtains the frame number from UI thread 1, if UI thread 1 has already started the animation() of the current frame, then UI thread 1 has already generated the frame number of the current frame. That is, the frame number currently held by UI thread 1 is the frame number of the current frame.

[0377] When UI thread 2 of the target application obtains the frame number from UI thread 1, if UI thread 1 has not yet started the animation() of the current frame, then UI thread 1 has not generated the frame number of the current frame. That is, the frame number currently held by UI thread 1 is the frame number of the previous frame.

[0378] S603, The UI thread 2 of the target application uses the obtained frame number as the frame number of the supplementary frame.

[0379] S604. After processing the data of frame Fx, and before sending the drawing instruction of frame Fx to the Render thread, the UI thread 1 of the target application sends an acknowledgment request to the UI thread 2 of the target application. The acknowledgment request carries the frame number of frame Fx.

[0380] The acknowledgment request carrying the frame number of frame Fx is used to request confirmation that a supplementary frame with the same frame number as frame Fx exists. Frame Fx is any image frame.

[0381] According to the logic of step S602 above, if UI thread 1 has started animation() for frame Fx before UI thread 2 performs frame interpolation, then the frame number obtained by UI thread 2 is the frame number of frame Fx, and the frame number of frame Fx is used as the frame number of the interpolated frame. Since the target position calculation time is very short, it can be assumed that the target position calculation is completed during animation(). Therefore, if UI thread 2 has an interpolated frame with the same frame number as frame Fx after UI thread 1 has finished processing frame Fx, it means that UI thread 2 performed frame interpolation before UI thread 1 finished processing the data of frame Fx, and UI thread 1 had already completed the target position calculation before interpolation.

[0382] If UI thread 1 does not start the animation() of frame Fx before UI thread 2 performs frame interpolation, then UI thread 2 will obtain the frame number of frame Fx-1, and use the frame number of frame Fx-1 as the frame number of the interpolated frame. Therefore, if UI thread 2 does not have an interpolated frame with the same frame number as frame Fx after UI thread 1 has processed frame Fx, it means that UI thread 2 did not perform frame interpolation before UI thread 1 finished processing the data of frame Fx, or that UI thread 2 performed frame interpolation, but UI thread 1 did not complete the calculation of the target position before the interpolation.

[0383] S605. In response to the confirmation request, UI thread 2 of the target application returns a confirmation result to UI thread 1 of the target application.

[0384] Optionally, the UI thread 1 of the target application can use preset values ​​to represent the confirmation result. For example, true indicates that there is a supplementary frame with the same frame number as frame Fx; false indicates that there is no supplementary frame with the same frame number as frame Fx.

[0385] S606. The UI thread 1 of the target application determines whether the confirmation result is true; if yes, proceed to step S607; if no, proceed to step S608.

[0386] S607, The UI thread 1 of the target application intercepts the rendering of frame Fx.

[0387] S608, The UI thread 1 of the target application sends the drawing instruction of frame Fx to the Render thread.

[0388] A confirmation result of true indicates the existence of a supplementary frame with the same frame number as frame Fx. This means that while UI thread 1 was processing the data for frame Fx, UI thread 2 performed frame supplementation, and UI thread 1 had already completed the calculation of the target position before supplementation. Therefore, it's necessary to intercept the rendering of frame Fx. Intercepting the rendering of frame Fx can be done, for example, by not sending the drawing instructions for frame Fx to the Render thread. This prevents the interface from reverting to its previous state and improves the user's visual experience.

[0389] If the confirmation result is false, it means that there is no supplementary frame with the same frame number as frame Fx. This indicates that UI thread 2 did not perform frame supplementation during the process of UI thread 1 processing the data of frame Fx, or UI thread 2 performed frame supplementation, but UI thread 1 did not complete the calculation of the target position before frame supplementation, so there is no need to intercept the rendering of frame Fx.

[0390] It should be noted that in the above embodiments, the target application is a third-party social application, the target view is a scrolling view, and the swipe operation is an upward swipe. In other embodiments, the target application can also be a system application, such as a control center or application management system. Moreover, the swipe operation can also be a left or right swipe.

[0391] For example, Figure 22 uses the recent tasks interface of an application management app as an example. The target view mentioned above can be the view corresponding to area 2201 in the figure, which is a scrolling view. The thumbnails of each application can be subviews in this scrolling view. The thumbnails can be swiped left and right through the root view (i.e., the container view) of this scrolling view. If the UI thread 1 of the application management app lags during the user's left and right swiping operation in the interface shown in Figure 22, the view corresponding to area 2201 in the entire figure can be shifted frame by frame according to the above process to simulate swiping and solve the swiping lag problem.

[0392] Alternatively, the target view may not be a scrolling view, but rather one or more fixed views within the interface. For example, Figure 23 uses the Control Center app as an example. When a user performs a leftward swipe operation on interface 2301 shown in Figure 23(a), switching from interface 2301 to interface 2302 shown in Figure 22(b), the various sub-views within region 2303 can be considered as a whole and used as the target view. If the UI thread 1 of the Control Center app experiences lag, the target view corresponding to region 2303 can be shifted frame by frame to simulate swiping, thus resolving the swiping lag issue. Of course, the process of switching from interface 2302 to interface 2301 is similar.

[0393] In addition, as a possible implementation, the functionality of UI thread 2 can also be implemented by other threads or modules, such as the Render thread or SurfacFlinger, etc., without limitation.

[0394] The foregoing has detailed examples of image processing methods provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0395] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0396] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0397] The electronic device provided in this embodiment is used to execute the above-described image processing method, and therefore can achieve the same effect as the above-described implementation method.

[0398] When using integrated units, the electronic device may further include a processing module, a storage module, and a communication module. The processing module is used to control and manage the operation of the electronic device. The storage module supports the execution of stored program code and data. The communication module supports communication between the electronic device and other devices.

[0399] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0400] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment can be a device having the structure shown in FIG3.

[0401] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the image processing method of any of the above embodiments.

[0402] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0403] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0404] Specifically, the apparatus provided in this application may include a first drawing module and a second drawing module. The first drawing module may correspond to the drawing module in the above embodiments. The second drawing module may correspond to the parallel frame interpolation module described above. The first drawing module is used to: draw a first image frame through a first UI thread of the target application, and trigger the rendering thread of the target application to render the first image frame through the first UI thread. The second drawing module is used to: in response to a sliding operation on the target interface of the target application, if the first UI thread experiences a lag during the sliding process, draw a second image frame through a second UI thread of the target application, and trigger the rendering thread to render the second image frame through the second UI thread; the second UI thread is different from the first UI thread.

[0405] In one embodiment, the target interface includes a slidable target view, and the second drawing module is specifically used to: update the position information of the target view in the second image frame through the second UI thread based on the sliding operation and the start time of the lag in the first UI thread.

[0406] In one embodiment, the second drawing module is specifically used to: determine, through the second UI thread, a first offset and a first offset direction of the target view relative to the lag start position; the lag start position is the position of the target view in the image frame corresponding to the lag start moment; the second drawing module is also used to: update the position information of the target view through the second UI thread according to the first offset and the first offset direction.

[0407] In one embodiment, the second drawing module is further configured to: if the first UI thread finishes lag, restore the target view to the position where the lag began via the second UI thread.

[0408] In one embodiment, the first UI thread experiences a stutter when the time taken for the first UI thread to draw one image frame exceeds a preset time.

[0409] In one embodiment, the first drawing module is further configured to: draw a third image frame through the first UI thread if the first UI thread lags; trigger the rendering thread to render the third image frame through the first UI thread; draw a fourth image frame through the first UI thread; and trigger the rendering thread to render the fourth image frame through the first UI thread.

[0410] In one embodiment, the second rendering module is further configured to: if the target interface does not stop scrolling when the first UI thread finishes lagging, send the first parameter of the third image frame to the rendering thread via the second UI thread; the first parameter includes the position parameter of the first region in the third image frame and the motion effect parameter corresponding to the first region, wherein the first region is the area in the third image frame with newly added display content compared to the last frame, the second image frame. The device also includes a rendering module, which is configured to render the third image frame based on the first parameter via the rendering thread.

[0411] In one embodiment, the first drawing module is further configured to: determine whether the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame; the fifth image frame is the image frame being drawn when the first UI thread experiences a lag; if the first UI thread has not completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread triggers the rendering thread to execute the rendering of the fifth image frame; if the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread intercepts the rendering of the fifth image frame.

[0412] In one embodiment, the second drawing module is further configured to: obtain the target frame number from the first UI thread when the first UI thread starts to lag via the second UI thread; the first drawing module is further configured to: send the target frame number to the second UI thread via the first UI thread, the target frame number being the frame number currently held by the first UI thread; if the first UI thread has already started executing the animation() function for the fifth image frame, then the target frame number is the number of the fifth image frame; if the first UI thread has not yet started executing the animation() function for the fifth image frame, then the target frame number is the frame number of the image frame preceding the fifth image frame; the second drawing module is further configured to: use the target frame number as the frame number of the second image frame via the second UI thread; the first drawing module is further configured to: if the frame number of the second image frame is different from that of the fourth image frame, then determine that the first UI thread has not completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame; if the frame number of the second image frame is the same as that of the fourth image frame, then determine that the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame.

[0413] In one embodiment, the device further includes a SurfaceFlinger, a rendering module, and a screen. The SurfaceFlinger is used to: composite a first image frame after the rendering thread of the target application is triggered by a first UI thread to render a first image frame; the screen is used to: display the first image frame; the SurfaceFlinger is also used to: composite a second image frame after the rendering thread of the target application is triggered by a second UI thread to render a second image frame; the screen is also used to: display the second image frame; the SurfaceFlinger is also used to: composite a third image frame after the rendering thread of the target application is triggered by a first UI thread to render a third image frame; the screen is also used to: display the third image frame; the SurfaceFlinger is also used to: composite a fourth image frame after the rendering thread of the target application is triggered by a first UI thread to render a fourth image frame; the screen is also used to: display the fourth image frame.

[0414] In one embodiment, the target interface includes a swipeable target view, and the swiping of the target interface is a hands-free swipe. The device also includes a lag detection module. The lag detection module is used to: determine whether the first UI thread starts to lag when it is determined that the target view starts to swipe away from the hands; the second drawing module is also used to: if the first UI thread starts to lag, draw at least one second image frame through the second UI thread; the lag detection module is also used to: determine whether the lag of the first UI thread has ended; the second drawing module is used to: if the lag of the first UI thread has ended, stop drawing the second image frame through the second UI thread.

[0415] In one embodiment, the second drawing module is further configured to: if the first UI thread lags during the sliding of the target interface, and the target application is determined to be an application in the application whitelist and the target interface includes a target view, then draw a second image frame through the second UI thread of the target application; the target view is a view in the view whitelist corresponding to the target application.

[0416] In one embodiment, the device further includes a view recognition module, which is configured to: determine whether the target interface after the target application jumps includes a target view when the target application interface jumps; if the target interface includes a target view, determine whether the process of the target application has completed initialization; initialization includes creating a second UI thread of the target application in the current process; if the process of the target application has not completed initialization, perform initialization.

[0417] In one embodiment, the view recognition module is specifically used to: determine whether the target interface after the target application jumps includes a target view when the target application's interface jumps, including: determining whether the target interface includes a target view when the target application's thread performs window layout or view layout.

[0418] The electronic device, computer-readable storage medium, computer program product, or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0419] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0420] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0421] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0422] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0423] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0424] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, the method being performed by an electronic device, characterized by, The method includes: The first image frame is drawn on the first UI thread of the target application. The first UI thread triggers the rendering thread of the target application to render the first image frame; In response to a swipe operation on the target interface of the target application, if the first UI thread lags during the swipe operation on the target interface, the second UI thread of the target application draws a second image frame; the second UI thread is different from the first UI thread. The second UI thread triggers the rendering thread to render the second image frame.

2. The method of claim 1, wherein, The target interface includes a scrollable target view, and the second UI thread of the target application draws a second image frame, including: The second UI thread updates the position information of the target view in the second image frame based on the swipe operation and the start time of the lag in the first UI thread.

3. The method of claim 2, wherein, The second UI thread updates the position information of the target view in the second image frame based on the swipe operation and the start time of the lag in the first UI thread, including: The second UI thread determines the first offset and the first offset direction of the target view relative to the lag start position based on the swipe operation; the lag start position is the position of the target view in the image frame corresponding to the lag start time. The second UI thread updates the position information of the target view based on the first offset and the first offset direction.

4. The method of claim 3, wherein, The method further includes: If the first UI thread finishes lag, the second UI thread will restore the target view to the position where the lag began.

5. The method according to any one of claims 1 to 4, characterized in that, The first UI thread experiences lag when the time taken for the first UI thread to draw one image frame exceeds a preset time.

6. The method according to any one of claims 1 to 5, characterized in that, After the second UI thread of the target application draws the second image frame, the method further includes: If the first UI thread stops lagging, then the first UI thread draws the third image frame; The first UI thread triggers the rendering thread to render the third image frame; The first UI thread draws the fourth image frame; The first UI thread triggers the rendering thread to render the fourth image frame.

7. The method according to claim 6, characterized in that, The method further includes: If the target interface does not stop scrolling when the first UI thread finishes lag, the second UI thread sends the first parameter of the third image frame to the rendering thread; the first parameter includes the position parameter of the first region in the third image frame and the motion effect parameter corresponding to the first region, the first region being the region in the third image frame that has added content to be displayed compared to the last frame of the second image frame; The rendering thread renders the third image frame based on the first parameter.

8. The method according to claim 6 or 7, characterized in that, The method further includes: Before the second UI thread starts drawing the second image frame, determine whether the first UI thread has completed the calculation of the view position in the fifth image frame; the fifth image frame is the image frame being drawn when the first UI thread experiences a lag. If the first UI thread has not completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread triggers the rendering thread to execute the rendering of the fifth image frame; If the first UI thread completes the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, then the first UI thread intercepts the rendering of the fifth image frame.

9. The method according to claim 8, characterized in that, Before determining whether the first UI thread has completed the calculation of the view position in the fifth image frame before the second UI thread starts drawing the second image frame, the process includes: When the first UI thread starts to lag, the second UI thread obtains the target frame number from the first UI thread; The first UI thread sends the target frame number to the second UI thread. The target frame number is the frame number currently held by the first UI thread. If the first UI thread has already started executing the animation() function for the fifth image frame, then the target frame number is the number of the fifth image frame. If the first UI thread has not yet started executing the animation() function for the fifth image frame, then the target frame number is the frame number of the image frame preceding the fifth image frame. The second UI thread uses the target frame number as the frame number of the second image frame; If the frame number of the second image frame is different from that of the fourth image frame, it is determined that the first UI thread did not complete the calculation of the view position in the fifth image frame before the second UI thread started drawing the second image frame; If the second image frame has the same frame number as the fourth image frame, then it is determined that before the second UI thread starts drawing the second image frame, the first UI thread completes the calculation of the view position in the fifth image frame.

10. The method according to any one of claims 6 to 9, characterized in that, After the first UI thread triggers the rendering thread of the target application to render the first image frame, the method further includes: SurfaceFlinger synthesizes the first image frame; The screen displays the first image frame; After the second UI thread triggers the rendering thread to render the second image frame, the method further includes: The SurfaceFlinger synthesizes the second image frame; The screen displays the second image frame; After the first UI thread triggers the rendering thread to render the third image frame, the method further includes: The SurfaceFlinger synthesizes the third image frame; The screen displays the third image frame; After the first UI thread triggers the rendering thread to render the fourth image frame, the method further includes: The SurfaceFlinger synthesizes the fourth image frame; The screen displays the fourth image frame.

11. The method according to any one of claims 1 to 10, characterized in that, The target interface includes a scrollable target view, and the scrolling of the target interface is a hands-free scrolling. During the scrolling of the target interface, if the first UI thread experiences a lag, the second UI thread of the target application draws a second image frame, including: When it is determined that the target view begins to slide off the hand, it is determined whether the first UI thread begins to lag; If the first UI thread starts to lag, the second UI thread will draw at least one frame of the second image. Determine if the lag in the first UI thread has ended; If the first UI thread stops lagging, the second UI thread stops drawing the second image frame.

12. The method according to any one of claims 1 to 11, characterized in that, If the first UI thread experiences a lag during the sliding process on the target interface, the second UI thread of the target application draws a second image frame, including: If the target application is determined to be an application in the application whitelist, and the target interface includes a target view, if the first UI thread lags during the scrolling of the target interface, the second UI thread of the target application draws a second image frame; the target view is a view in the view whitelist corresponding to the target application.

13. The method according to claim 12, characterized in that, The method further includes: When the target application's interface changes, determine whether the target view is included in the target interface after the change. If the target interface includes the target view, then it is determined whether the process of the target application has completed initialization; the initialization includes creating the second UI thread of the target application in the current process; If the process of the target application has not completed the initialization, then the initialization is performed.

14. The method according to claim 12 or 13, characterized in that, When the target application's interface changes, determining whether the target view is included in the target interface after the change includes: When the target application's thread performs window layout or view layout, determine whether the target interface includes the target view.

15. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 14.

16. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 14.

Images