Image synthesis method, electronic device, storage medium, chip system, and computer program
By using a hardware hybrid renderer (HWC) in electronic devices and utilizing a blur buffer to store blurred layers, the problem of interface flickering during layer overlay was solved, resulting in a lower power consumption and smoother image compositing process.
Patent Information
- Application Number
- PCT/CN2025/095549
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2026-11-19
AI Technical Summary
In electronic devices, the blurring effect caused by layer overlays, resulting in flickering due to changes in interface brightness, negatively impacts the user experience, especially noticeable during volume adjustment or message pop-ups.
During image compositing, the Hardware Hybrid Renderer (HWC) is used to maintain the continuity of HWC compositing by avoiding the forced use of GPU compositing during blur processing. The blur buffer stores the blur layers and updates them when necessary, reducing the need to switch compositing modes.
It reduces the power consumption, load, and rendering latency of electronic devices in real-time blurred scenes, reduces flickering caused by brightness changes, and improves the user experience.
Smart Images

Figure CN2025095549_19112026_PF_FP_ABST
Abstract
Description
Image synthesis methods, electronic devices, storage media, chip systems, and computer programs Technical Field
[0001] This application relates to the field of terminal technology, and in particular to image synthesis methods, electronic devices, storage media, chip systems, and computer programs. Background Technology
[0002] With the development of terminal technology, electronic devices are increasingly demanding more image processing capabilities. Electronic devices can use hardware compositors (HWC) and / or graphics processing units (GPUs) to composite images with multiple layers. To provide users with more visually appealing information, electronic devices can blur some layers when multiple layers overlap.
[0003] In a possible implementation, the screen of an electronic device may flicker in real-time blurred scenarios such as adjusting volume or displaying message pop-ups. For example, when receiving an operation to adjust the volume or an instruction to display a message pop-up, the screen brightness of the electronic device dims; after canceling the display of the volume bar or message pop-up, the screen brightness of the electronic device returns to its previous level; the changes in brightness of the screen on the electronic device will affect the user experience. Summary of the Invention
[0004] This application provides an image compositing method, an electronic device, a storage medium, a chip system, and a computer program, applicable to the field of terminal technology. Image compositing is performed using HWC before or after the volume bar, message pop-up, or sidebar appears; HWC compositing is also used during the display of the volume bar, message pop-up, or sidebar. The electronic device uses HWC compositing throughout the entire process, avoiding the brightness changes in the interface caused by forcibly using GPU compositing during blur processing.
[0005] In a first aspect, embodiments of this application propose an image compositing method applied to an electronic device, the electronic device including a Hardware Hybrid Renderer (HWC). The method includes: displaying a first interface (e.g., the interface shown in Figure 3a or Figure 4a), the first interface including a first layer (e.g., a social application window layer or a video application window layer), and excluding a blurred layer, the size of the first layer being the same as the screen size of the electronic device; the first interface being composed by HWC; receiving a first operation (e.g., clicking a volume button) in the first interface; and displaying a second interface (e.g., the interface shown in Figure 3b to the interface shown in Figure 3e, or the interface shown in Figure 4b to the interface shown in Figure 4e) according to the first operation. The second interface includes a first layer (e.g., involving an application window layer or a video application window layer), a second layer (e.g., a volume bar layer), and a blur layer corresponding to the second layer; the blur layer is obtained from a blur buffer; when the second layer is a preset layer and carries blur parameters, the second interface is composited by HWC of the first layer, the second layer, and the blur layer; the preset layer includes any of the following: a volume bar layer, a message pop-up layer, or a sidebar layer; the blur layer is located above the first layer, and the second layer is located above the blur layer; after the second layer disappears, the first interface is displayed (e.g., the interface shown in f in Figure 3 or the interface shown in f in Figure 4).
[0006] This process corresponds to the embodiment shown in Figure 16. In non-real-time blurred scenes, the electronic device uses the native HWC compositing workflow to obtain the first interface; in real-time scenes, the electronic device can obtain the blurred layer from the stable blurred buffer and then use HWC to composite the second interface. Throughout the switching process between non-real-time blurred scenes, real-time blurred scenes, and non-real-time blurred scenes, the electronic device continuously uses HWC compositing.
[0007] In this way, the electronic device uses HWC compositing when the target layer is not displayed; the electronic device can still use HWC compositing when the target layer is displayed, reducing the switching process between HWC compositing and GPU compositing; it can reduce the power consumption, load and rendering latency of the electronic device when forced to use GPU compositing in real-time blurred scenes; at the same time, it reduces the flickering problem of light-dark-light caused by switching compositing methods.
[0008] In one possible implementation, the electronic device further includes a graphics processing unit (GPU); the second interface includes a first image; displaying the second interface includes: acquiring multiple layers of the first image, the multiple layers of the first image including a first layer and a second layer of the first image; the GPU blurring the first layer of the first image according to blur parameters to obtain a blurred layer of the first image; the blurred layer of the first image is stored in a blur buffer; the HWC composites the blurred layer of the first image, the first layer of the first image, and the second layer of the first image to obtain the first image; and displaying the first image.
[0009] This process corresponds to the embodiment shown in Figure 8. During the process of synthesizing the copy frame image, the electronic device can generate a blurred layer of the copy frame image and store it for later use.
[0010] In this way, the blur layer generated by the GPU can be stably stored in the blur buffer, so that the HWC compositing method can be used during the image compositing process, and the blur layer can be reused when compositing adjacent frame images; thereby reducing the power consumption, load and rendering latency of electronic devices when the GPU is forced to compose in real-time blur scenes; at the same time, reducing the flickering problem of bright-dark-bright caused by switching compositing methods.
[0011] In one possible implementation, the second interface further includes a second image, which is the next frame of the first image; after displaying the first image, the interface further includes: acquiring multiple layers of the second image, including a first layer of the second image and a second layer of the second image; if the size or position of the second layer of the second image and the second layer of the first image are different, and the screen refresh rate is greater than the refresh rate threshold, the blurred layer of the first image in the blur buffer is not updated; the blurred layer of the first image is stored in the blur buffer; or, if the screen refresh rate is less than or equal to the refresh rate threshold, the GPU performs blur processing on the first layer of the second image according to the blur parameters to obtain the blurred layer of the second image; and the blurred layer of the second image is updated in the blur buffer.
[0012] This process corresponds to the embodiment shown in Figure 12. When the second layer is in the changing phase (as shown in interface b in Figure 3, interface e in Figure 3, interface b in Figure 4, and interface e in Figure 4), the electronic device can determine whether to update the blur layer based on the screen refresh rate. In high refresh rate scenarios, the electronic device can alternate between using copied frame images and skipped frame images; in low refresh rate scenarios, the electronic device continuously uses copied frames.
[0013] In this way, during the target layer pop-up and collapse phases, electronic devices can reuse the blurred layer of the previous frame image by skipping frames, reducing the number of times the GPU is called to generate the blurred layer, thus reducing GPU power consumption and load; and by using HWC compositing, brightness changes caused by switching compositing methods can be avoided.
[0014] In one possible implementation, the second interface further includes a third image, where the first image is the a-th frame, the second image is the (a+1)-th frame, and the third image is the b-th frame, where a is less than b, and both a and b are positive integers. The first to b-1 frames are composited by HWC using the blurred layer of the first image. After not updating the blurred layer of the first image in the blur buffer, the interface further includes: acquiring multiple layers of the third image, including a first layer and a second layer of the third image; if the second layer of the third image and the second layer of the first image have different sizes or positions, and the screen refresh rate is greater than a refresh rate threshold, determining whether the number of times the blurred layer of the first image has been used has reached a first-time threshold; if the first-time threshold has been reached, the GPU blurs the first layer of the third image according to blur parameters to obtain a blurred layer of the third image; the blurred layer of the third image is updated in the blur buffer; HWC composites the blurred layer, the first layer, and the second layer of the third image to obtain the third image; and the third image is displayed.
[0015] This process corresponds to the embodiment shown in Figure 12a. The electronic device can use one or more frames to skip frames after copying a frame; when the number of times reaches a threshold, the electronic device uses copied frames to update the blurred layer.
[0016] In this way, during the target layer pop-up and target layer collapse phases, the electronic device can reuse the blurred layer of the previous frame image through the M-frame skipping image, reducing the number of times the GPU is called to generate the blurred layer, thus reducing GPU power consumption and load. At the same time, after a period of time (e.g., when the number of times reaches a threshold), the blurred layer is updated to improve the similarity and accuracy between the blurred layer of the previous frame copy image and the blurred layer of the current frame image.
[0017] In one possible implementation, the second interface further includes a fourth image, where the first image is the c-th frame image, the fourth image is the d-th frame image, c is less than d, and both c and d are positive integers; the second layer in the (c+1)-1 to (d-1)-1-th frames has the same size and position as the second layer of the first image; the first layer in the (c+1)-1 to (d-1)-th frames has the same image as the first layer of the first image; the (c+1)-1 to (d-1)-th frames are all composited by HWC using the blurred layer of the first image; after displaying the first image, the interface further includes: obtaining multiple layers of the fourth image, including the first layer and the second layer of the fourth image; when the second layer of the fourth image has the same size or position as the second layer of the first image, and the first layer of the fourth image has a different image, the GPU blurs the first layer of the fourth image according to the blur parameters to obtain the blurred layer of the fourth image; and updating the blurred layer of the fourth image in the blur buffer.
[0018] This process corresponds to the embodiment shown in Figure 13c. In the stable phase (e.g., the interface shown in Figure 3c and Figure 3d), and the lower-level interface layer no longer changes (e.g., the screen of the social application window in Figure 3c and Figure 3d does not change), the electronic device can continue using the skipped-frame image until the lower-level interface layer changes, at which point the electronic device updates the blurred layer using a copied frame.
[0019] In this way, when the volume bar is stable and the lower interface has not changed, the electronic device can set up multi-frame skipping images. All multi-frame skipping images reuse the blur layer of the previous frame copy image, further reducing the number of times the GPU is called to generate blur layers, thus reducing GPU power consumption and load.
[0020] In one possible implementation, the second interface further includes a fifth image, which is the next frame of the first image. After displaying the first image, the interface further includes: acquiring multiple layers of the fifth image, including a first layer and a second layer of the fifth image; when the second layer of the fifth image and the second layer of the first image have the same size or position, and the first layer of the fifth image and the first layer of the first image have different images, HWC composites the blurred layer of the first image, the first layer of the fifth image, and the second layer of the fifth image to obtain the fifth image; the blurred layer of the first image is stored in the blur buffer; and the fifth image is displayed.
[0021] This process corresponds to the embodiment shown in Figure 13a or Figure 13b. In a stable phase (e.g., the interface shown in Figure 4c and Figure 4d), and where the lower-level interface layer is constantly changing (e.g., the video application window in Figure 4c and Figure 4d changes), the electronic device can alternate between copy frames and skipped frames. For example, when the previous frame was a copy frame, the current frame could be a skipped frame.
[0022] In this way, when the volume bar is stable and the underlying interface changes, the electronic device can reduce the number of times the GPU is called to generate a blur layer by adjusting the ratio of the skip frame image to the copy frame image, thereby reducing GPU power consumption and load. In addition, even if the underlying interface changes, reusing the blur layer of the copy frame in the skip frame image will not affect the smoothness of the user's viewing of the frame image.
[0023] In one possible implementation, the second interface further includes a sixth image, where the first image is the e-th frame image, the sixth image is the f-th frame image, e is less than f, and both e and f are positive integers; the first image to the (f-1)-th frame images are all composited by HWC using the blurred layer of the first image; after displaying the fifth image, the interface further includes: obtaining multiple layers of the sixth image, including the first layer and the second layer of the sixth image; if the second layer of the sixth image and the second layer of the first image have the same size or position, and the first layer of the sixth image and the first layer of the first image have different images, determining whether the number of times the blurred layer of the first image has been used has reached a second threshold; if it is determined that the second threshold has been reached, the GPU blurs the first layer of the sixth image according to the blur parameters to obtain the blurred layer of the sixth image; the blurred layer of the sixth image is updated in the blur buffer; HWC composites the blurred layer, the first layer, and the second layer of the sixth image to obtain the sixth image; and the sixth image is displayed.
[0024] This process corresponds to the embodiments shown in Figure 13a or Figure 13b. In a stable phase (e.g., the interfaces shown in Figure 4c and 4d), and where the lower-level interface layer is constantly changing (e.g., the video application window in interfaces 4c and 4d changes), the electronic device can alternate between copying frames and skipping frames. For example, when the previous frame is a skipped frame, it can be determined whether the required number of skips has been reached; if so, the current frame uses a copy of the frame to update the blurred layer.
[0025] In this way, during the target layer stabilization phase, the electronic device can reuse the blurred layer of the previous frame image through N-hop frame images, reducing the number of times the GPU is called to generate the blurred layer, thus reducing GPU power consumption and load. At the same time, after a period of time (e.g., when the number of times reaches a threshold), the blurred layer is updated to improve the similarity and accuracy between the blurred layer of the previous frame image and the blurred layer of the current frame image.
[0026] In one possible implementation, before displaying the second interface, the method further includes: in response to the first operation, acquiring multiple layers of the seventh image; the image of the first interface is the g-th frame image, and the seventh image is the g+1-th frame image, where g is a positive integer; the multiple layers of the seventh image include a first layer and a second layer of the seventh image; under the condition of satisfying a first preset condition, creating a blur buffer; simultaneously, masking the blur parameters, and using HWC to composite the first layer and the second layer of the seventh image to obtain the seventh image; wherein, during the process of compositing the seventh image, the blur buffer does not include the blur layer; the first preset condition includes: the second layer carries blur parameters, the second layer is a preset layer, the number of second layers is 1, and the first layer is not a safe layer.
[0027] This process corresponds to steps S1402 and S1403 in Figure 14. Before displaying the copied frame and skipped frames, the electronic device displays a normal frame image. During the processing of the normal frame image, the electronic device can simultaneously create a blur buffer for subsequent storage of the blur layer.
[0028] In this way, the electronic device can be controlled to transition from a non-real-time blurred scene to a real-time blurred scene through normal frame images. A blur buffer layer is created in the normal frame state for subsequent HWC synthesis of the real-time blurred scene image. In addition, by masking the blur parameters, the normal frame is also controlled to use HWC synthesis, so that the electronic device uses HWC synthesis throughout the entire process from non-real-time blurred scene to real-time blurred scene, thereby avoiding flickering problems caused by inconsistent brightness.
[0029] In one possible implementation, after receiving the first operation in the first interface, the method further includes: displaying a third interface according to the first operation; the third interface includes a first layer, a second layer, and a blurred layer corresponding to the second layer; the blurred layer is obtained from a temporary buffer of the GPU; if multiple layers of the third interface satisfy a second preset condition, the third interface is composited by the GPU from the multiple layers of the third interface; the second preset condition includes one or more of the following: the second layer carries a blur parameter, and the number of second layers is greater than 1; or, the second layer carries a blur parameter, and the preset layer does not include the second layer; or, the second layer carries a blur parameter, and the first layer is a safe layer.
[0030] In some blurry scenes, electronic devices can use native GPU compositing to avoid issues such as insufficient computing power for HWC compositing or reduced security of the security layer due to screenshots.
[0031] Secondly, this application proposes an image compositing method applied to an electronic device, which includes a Hardware Hybrid Renderer (HWC). The method includes: acquiring multiple layers of a first interface; the multiple layers of the first interface include a first layer but do not include a blurred layer, and the size of the first layer is the same as the screen size of the electronic device; the HWC composites the multiple layers of the first interface to obtain the first interface; displaying the first interface; receiving a first operation in the first interface; acquiring multiple layers of a second interface; the multiple layers of the second interface include a first layer and a second layer; determining whether the multiple layers of the second interface meet a first preset condition; the first preset condition includes that the second layer carries a blurred parameter, the second layer is a preset layer, the number of second layers is 1, and the first layer is not a safe layer; the preset layer includes any one of the following: a volume bar layer, a message pop-up layer, or a sidebar layer; when the first preset condition is met, from The blur layer corresponding to the second layer is obtained from the blur buffer; HWC composites the first layer of the second interface, the second layer of the second interface, and the blur layer of the second interface to obtain the second interface; the blur layer is located above the first layer, and the second layer is located above the blur layer; or, if the first preset condition is not met and the second preset condition is met, the GPU blurs the first layer of the second interface to obtain the blur layer; the blur layer is stored in a temporary buffer; the second preset condition includes one or more of the following: the second layer carries blur parameters, and the number of second layers is greater than 1; or, the second layer carries blur parameters, and the preset layers do not include the second layer; or, the second layer carries blur parameters, and the first layer is a safe layer; the GPU composites the first layer of the second interface, the second layer of the second interface, and the blur layer of the second interface to obtain the second interface; the second interface is displayed.
[0032] This process can be described in the embodiment shown in Figure 17.
[0033] In this way, the electronic device uses HWC compositing when the target layer is not displayed; the electronic device can still use HWC compositing when the target layer is displayed, reducing the switching process between HWC compositing and GPU compositing; it can reduce the power consumption, load and rendering latency of the electronic device when forced to use GPU compositing in real-time blurred scenes; at the same time, it reduces the flickering problem of light-dark-light caused by switching compositing methods.
[0034] In one possible implementation, the second interface includes a first image and a second image; the method includes: after obtaining the first layer and the second layer of the first image, the GPU performs blur processing on the first layer of the first image according to blur parameters to obtain a blurred layer of the first image; the blurred layer of the first image is stored in a blur buffer; the blurred layer of the first image is obtained from the blur buffer; the HWC composites the first layer, the second layer, and the blurred layer of the first image to obtain the first image; after obtaining the first layer and the second layer of the second image, the blurred layer of the first image is obtained from the blur buffer; the HWC composites the first layer, the second layer, and the blurred layer of the first image to obtain the second image.
[0035] The first image can be a copied frame image. The electronic device can generate a blur layer during the synthesis of the first image and store the blur layer in a blur buffer. The second image can be a skipped frame image. The electronic device can reuse the blur layer of the first image (a copied frame from the previous frame) during the synthesis of the second image.
[0036] In this way, the electronic device can obtain a stable blurred layer through the blur buffer, enabling the synthesis of real-time blurred scene images via HWC and the reuse of blurred layers. When the blurred layers of two adjacent frames are the same or similar, the electronic device can skip the GPU blur processing. The electronic device uses the blurred layer of the previous frame as the blurred layer of the current frame, reducing the number of GPU blur processing operations and lowering GPU power consumption and load by reusing blurred layers. In addition, the electronic device uses HWC to synthesize multiple layers in the second frame image with the blurred layer, which reduces the forced GPU synthesis process for real-time blurred scenes, thereby reducing the bright-dark-bright flickering problem caused by switching synthesis methods.
[0037] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory is used to store computer execution instructions, and the processor is used to run the computer execution instructions stored in the memory to perform the method described in the first aspect or any possible implementation of the first aspect.
[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.
[0039] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run, causes the computer to perform the method described in the first aspect or any possible implementation thereof.
[0040] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.
[0041] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).
[0042] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description
[0043] Figure 1 is a schematic diagram of an interface for a possible real-time blurred scene;
[0044] Figure 2 is a flowchart illustrating one possible image synthesis method.
[0045] Figure 3 is a schematic diagram of an interface for a real-time blurred scene provided in an embodiment of this application;
[0046] Figure 4 is a schematic diagram of another real-time blurred scene interface provided in an embodiment of this application;
[0047] Figure 5 is a schematic diagram of the interface for various real-time blurred scenes provided in the embodiments of this application;
[0048] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0049] Figure 7 is a software structure block diagram of an electronic device provided in an embodiment of this application;
[0050] Figure 8 is a schematic flowchart of an image synthesis method provided in an embodiment of this application;
[0051] Figure 9 is a schematic flowchart of a GPU blurring process provided in an embodiment of this application;
[0052] Figure 10 is a flowchart illustrating another image synthesis method provided in an embodiment of this application;
[0053] Figure 11 is a schematic diagram of the change of a blurred layer provided in an embodiment of this application;
[0054] Figure 12 is a schematic diagram of the arrangement of copy frame images and skip frame images at different refresh rates provided in the embodiments of this application;
[0055] Figure 13 is a schematic diagram of the arrangement of copy frame images and skip frame images in the static interface and dynamic interface provided in the embodiments of this application;
[0056] Figure 14 is a flowchart illustrating an image synthesis method provided in an embodiment of this application;
[0057] Figure 15 is a schematic diagram of the arrangement of copy frame images and skip frame images in the volume adjustment scenario provided in the embodiment of this application;
[0058] Figure 16 is a flowchart illustrating an image synthesis method provided in an embodiment of this application;
[0059] Figure 17 is a flowchart illustrating another image synthesis method provided in an embodiment of this application;
[0060] Figure 18 is a schematic diagram of an image synthesis device provided in an embodiment of this application. Detailed Implementation
[0061] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:
[0062] 1. Real-time blurring
[0063] Real-time blurring is a technique in computer graphics and image processing used to blur dynamic images or video streams. This technique can be used in various applications, including but not limited to: in video, real-time blurring can be used to hide user information and protect personal privacy; in games or movies, real-time blurring can be used to create specific visual effects to enhance the dynamism and realism of the scene. Implementing real-time blurring typically requires efficient algorithms and powerful computing capabilities to ensure that no significant latency is introduced when processing dynamic content. Commonly used techniques include convolution operations, Fourier transforms, and GPU-based parallel computing.
[0064] 2. Electronic equipment
[0065] The electronic devices in this application embodiment may include handheld devices, vehicle-mounted devices, etc., with real-time blurring functionality. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.
[0066] By way of example and not limitation, in this embodiment, the electronic device may also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that are intelligently designed and developed using wearable technology, such as glasses, gloves, watches, clothing, and shoes.
[0067] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.
[0068] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.
[0069] 3. Other terms
[0070] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.
[0071] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0072] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.
[0073] In a possible implementation, the electronic device can use HWC and / or a GPU to composite images across multiple layers. The GPU has greater computing power than the HWC; it can support more complex hybrid computations. For example, in real-time blurred scenes, the electronic device can generate a blur patch using the GPU and composite the blur patch with multiple layers. In non-real-time blurred scenes, the electronic device can composite images across multiple layers using HWC.
[0074] Taking a mobile phone as an electronic device and a volume adjustment scenario as a real-time blurred scene as an example, the following description, in conjunction with Figure 1, illustrates the possible implementations of real-time blurred scenes:
[0075] When the screen is on, the electronic device can display the interface shown in Figure 1a; the interface shown in Figure 1a may include: desktop 101, status bar 102 and desktop wallpaper 103.
[0076] Desktop 101 may include time information and multiple application icons, etc.; the time information may include, for example, clock information and date information. Desktop 101 may also include elements such as service cards, but this embodiment of the application does not impose any limitations.
[0077] The status bar 102 can be located at the top of the electronic device, for example, it can be placed on the side where the front-facing camera is located. The status bar 102 is used to display the current status of the phone, such as mobile communication network signal quality indicator, wireless network signal quality indicator, battery status, clock information, etc.
[0078] Desktop wallpaper 103 can be a static wallpaper or a live wallpaper. Desktop wallpaper 103 may include, for example, a star pattern; live wallpaper may feature dynamic effects such as twinkling or moving stars. Desktop wallpaper 103 may be located on top of desktop 101.
[0079] In the interface shown in Figure 1a, when a trigger operation is received for the volume buttons, the electronic device can display the interface shown in Figure 1b. In the interface shown in Figure 1b, the electronic device can display the desktop 101, status bar 102, desktop wallpaper 103, volume bar 104, blur block 105, and volume adjustment control 106, etc.
[0080] Desktop 101, status bar 102, and desktop wallpaper 103 can be referred to in the relevant descriptions of the interface shown in Figure 1a.
[0081] The volume bar 104 can be used to adjust the volume. The volume bar 104 can consist of two parts; for example, the volume bar includes a foreground image 1041 and a background image 1042. The foreground image 1041 can be used to display the volume level; the background image 1042 can be a rounded rectangle, with a corner radius equal to half the width of the rounded rectangle. For example, the background image 1042 can be presented as a vertical capsule shape. The height of the background image is greater than or equal to the height of the foreground image, and the width of the background image is equal to the width of the foreground image.
[0082] Blur block 105 can be an image of the area corresponding to volume bar 104 in desktop 101 and desktop wallpaper 103 after blurring; the size of blur block 105 is the same as the size of volume bar background image 1041. For example, the area corresponding to volume bar 104 in desktop 101 and desktop wallpaper 103 can be area 1011; area 1011 can include stars and some time information from the desktop wallpaper. After blurring area 1011, blur block 105 is obtained; blur block 105 displays blurred stars and blurred part of the time information.
[0083] The volume control 106 can be used to display a volume control panel; users can use the volume control panel to quickly adjust, for example, alarm volume, notification volume, incoming call volume and / or media volume.
[0084] It should be noted that Figure 1b shows an interface that may include multiple layers; for example, the volume bar 104 may be located above the blur block 105, the blur block 105 may be located above the desktop 101, and the desktop 101 may be located above the desktop wallpaper 103. The electronic device can blur the corresponding area of the desktop 101 through the blur block 105. The volume bar background image 1041 can be set to a transparent or semi-transparent layer to present the blur effect of the desktop 101 and desktop wallpaper 103 through the blur block 105, thereby linking the volume bar 104 with the lower layer interface; wherein, the lower layer interface can be the interface below the volume bar 104 (such as the desktop 101 and desktop wallpaper 103). The linkage effect can be, for example, the stars in the desktop wallpaper showing dynamic effects such as movement and blinking, and the electronic device can present the dynamic effect of the stars moving and blinking in the lower desktop wallpaper in the corresponding area of the volume bar through the blur block. The electronic device can set the volume bar foreground image 1042 to an opaque image to clearly reflect the current volume value.
[0085] It should also be noted that, in possible implementations, the computing power of a GPU is greater than that of HWC. For complex hybrid computations, electronic devices can use GPU compositing methods; for example, the electronic device generates blur blocks during GPU compositing and then composites these blur blocks with other layers. For non-real-time blur scenes, electronic devices can use HWC compositing methods.
[0086] The interface shown in Figure 1a does not include blur blocks. The interface shown in Figure 1a corresponds to a non-real-time blurred scene. The electronic device can synthesize and render the interface shown in Figure 1a using HWC. The interface shown in Figure 1b includes blur blocks. The interface shown in Figure 1b corresponds to a real-time blurred scene. The electronic device can synthesize and render the interface shown in Figure 1b using GPU.
[0087] During the transition from the interface shown in Figure 1a to the interface shown in Figure 1b, the electronic device switches from HWC compositing to GPU compositing. There is a difference in color gamut mapping between HWC compositing and GPU compositing, resulting in a shift in their gamma values. This causes the brightness of the HWC-composited image to be higher than that of the GPU-composited image. For example, the brightness of the interface shown in Figure 1a is greater than that of the interface shown in Figure 1b (e.g., represented by dotted filling to indicate a decrease in brightness).
[0088] After volume adjustment ends (e.g., no trigger operation is received for the volume buttons within a preset time, or a trigger operation is received for other interface elements within a preset time), the electronic device can de-display the volume bar 104, the blur block 105, and the volume adjustment control 106, and the electronic device displays the interface shown in Figure 1c. The interface shown in Figure 1c includes the desktop 101, the status bar 102, and the desktop wallpaper 103; the desktop 101, the status bar 102, and the desktop wallpaper 103 can be referred to the relevant descriptions in the interface shown in Figure 1a.
[0089] It is understood that the interface shown in Figure 1c does not include the blur blocks. The interface shown in Figure 1c may correspond to a non-real-time blur scene. Electronic devices can obtain the interface shown in Figure 1c by compositing and rendering with HWC.
[0090] During the transition from the interface shown in Figure 1b to the interface shown in Figure 1c, the electronic device switches from GPU-synthesized images to HWC-synthesized images. Because the brightness of the HWC-synthesized image is higher than that of the GPU-synthesized image, the brightness of the interface shown in Figure 1b is lower than that of the interface shown in Figure 1c.
[0091] Before and after the volume bar is displayed, the electronic device may appear to dim and then brighten; users may see the electronic device's interface flickering between bright and dark, affecting the user experience.
[0092] The following describes the processing flow of real-time and non-real-time blurred scenes in possible implementations with reference to Figure 2; wherein, Figure 2a corresponds to the processing flow of the interface shown in Figure 1a or the interface shown in Figure 1c; Figure 2b corresponds to the processing flow of the interface shown in Figure 1b.
[0093] For non-real-time blurred scenes, please refer to Figure 2a:
[0094] During the display of the interface shown in Figure 1a or the interface shown in Figure 1c, the electronic device can obtain the desktop layer 101, the status bar layer 102, and the desktop wallpaper layer 103.
[0095] Electronic devices can transfer the Desktop 101 layer, Status Bar 102 layer, and Desktop Wallpaper 103 layer to HWC for HWC compositing. HWC can then composite these layers into a buffer. Specifically, HWC can first draw the Desktop Wallpaper 103 layer onto the buffer, then draw the Desktop 101 layer onto the buffer containing the Desktop Wallpaper 103 layer, and finally draw the Status Bar 102 layer onto the buffer containing both the Desktop 101 and Desktop Wallpaper 103 layers. The Desktop Wallpaper 103 layer is located below the Desktop 101 layer, and the Desktop 101 layer is located below the Status Bar 102 layer.
[0096] It should be noted that hierarchical and positional relationships may exist between the multiple layers in the embodiments of this application. Positional relationships can be used to describe the relative positions of two or more adjacent layers within the plane where the layers are located; for example, the blurred block 105 and the volume bar background image 1041 are located in the same position within the plane where the layers are located. Hierarchical relationships can be used to describe the vertical relationship between multiple layers in the layer normal direction; for example, the blurred block 105 is located below the volume bar 104; the lower layer is relatively closer to the buffer, and the upper layer is relatively farther from the buffer.
[0097] The electronic device renders and displays the buffer; the electronic device can display, for example, the interface shown in Figure 1a or the interface shown in Figure 1c.
[0098] For real-time blurred scenes, please refer to Figure 2b:
[0099] During the display of the interface shown in Figure 1b, the electronic device can obtain the desktop layer 101, the status bar layer 102, the desktop wallpaper layer 103, and the volume bar layer 104, etc.
[0100] Electronic devices can perform GPU compositing on graphics related to real-time blurring. For example, the electronic device passes a Desktop 101 layer, a Desktop Wallpaper 103 layer, and a Volume Bar 104 layer to the GPU for GPU compositing. The GPU can allocate a buffer and sequentially overlay (composite) the Desktop Wallpaper 103 layer and the Desktop 101 layer onto the buffer. The GPU then draws the overlaid Desktop Wallpaper 103 layer and the Desktop 101 layer onto a temporary buffer. The GPU blurs the Desktop Wallpaper 103 layer and the Desktop 101 layer in the temporary buffer, resulting in blurred Desktop Wallpaper 103 layers and Desktop 101 layers. The GPU then cuts out the area in the blurred Desktop Wallpaper 103 layer and the Desktop 101 layer that corresponds to the Volume Bar 104 layer, resulting in blurred block 105. The GPU draws blurred block 105 from the temporary buffer onto the buffer containing the Desktop Wallpaper 103 layer and the Desktop 101 layer, and then clears the temporary buffer. The GPU then draws the Volume Bar 104 layer onto the buffer containing blurred block 105, Desktop Wallpaper 103 layer, and Desktop 101 layer. After GPU compositing, the Desktop Wallpaper 103 layer is located below the Desktop 101 layer, and the Desktop 101 layer is located below the Blur Block 105; the Blur Block 105 is located below the Volume Bar 104 layer.
[0101] Electronic devices can perform HWC compositing on graphics related to non-real-time blur. For example, an electronic device passes the status bar 102 layer and the GPU-composited layer to HWC for HWC compositing. HWC can composite the status bar 102 layer and the GPU-composited layer into a buffer. Specifically, HWC can first draw the GPU-composited layer onto the buffer according to the hierarchy, and then draw the status bar 102 layer onto the buffer containing the GPU-composited layer, with the GPU-composited layer located below the status bar 102 layer.
[0102] The electronic device renders and displays the buffer; the electronic device can display an interface, for example, as shown in Figure 1b.
[0103] As can be seen, in non-real-time blurred scenes, the electronic device uses High-Wave Compatibility (HWC) to composite multiple layers; in real-time blurred scenes, the electronic device uses the GPU to composite multiple layers. That is, from the interface shown in Figure 1a to the interface shown in Figure 1c, the electronic device can experience a switching process between HWC compositing, GPU compositing, and HWC compositing. Therefore, in this scenario, the HWC-composited image will appear bright while the GPU-composited image will appear dark, causing the user to potentially see a flickering bright-dark-bright effect on the electronic device's interface. Furthermore, the GPU consumes more power than HWC; switching to GPU compositing increases GPU load, power consumption, and rendering latency, impacting the user experience.
[0104] In view of this, this application provides an image compositing method that creates a blur buffer (stage layer). The properties of the blur buffer are the same as those of the buffer used to cache layers such as the volume bar layer, the lower interface layer, and the status bar layer, meaning the blur buffer can exist stably. The electronic device obtains blur blocks through GPU blur processing and adds them to the blur buffer to obtain a blur layer. The size and position of the blur layer are the same as those of the volume bar layer, and the blur layer can be reused. Subsequently, in real-time blurred scenes, the electronic device can composite the blur layer, volume bar layer, lower interface layer, and status bar layer using HWC to obtain the image to be displayed. When the lower interface layer remains unchanged for multiple frames, the electronic device can reuse the blur layer for HWC compositing. Thus, in real-time blurred scenes, the electronic device can use HWC for image compositing. This process does not involve switching between HWC and the GPU, reducing the power consumption, load, and rendering latency of the electronic device. The reuse of blur layers reduces the number of times the GPU generates blur blocks, further reducing the power consumption, load, and rendering latency of the electronic device. Without involving HWC and GPU switching, electronic devices will not exhibit a bright-dark-bright flickering phenomenon, thus improving the user experience.
[0105] Taking the display of a social application interface on an electronic device as an example, Figure 3 shows a schematic diagram of the interface of an image synthesis method provided in an embodiment of this application. In the embodiment shown in Figure 3, the lower layer interface can be a static interface; as shown in Figure 3:
[0106] For example, when the screen is on, the electronic device can display the interface shown in Figure 3a; the interface shown in Figure 3a may include: a social application window 301 and a status bar 302. The social application window 301 includes multiple chat messages and text input boxes, and the status bar 302 can be referred to the relevant description of status bar 102, which will not be repeated here.
[0107] In the interface shown in Figure 3a, when a trigger operation is received on the volume button, the electronic device can display the interface shown in Figure 3b. The interface shown in Figure 3b can be the volume bar pop-up stage. In the interface shown in Figure 3b, the electronic device can display a social application window 301, a status bar 302, a volume bar 303, a blurred block 304, and a volume adjustment control 305, etc. The status bar 302, volume bar 303, blurred block 304, and volume adjustment control 305 can be referred to the relevant descriptions in the interface shown in Figure 1a, and will not be repeated here.
[0108] The blurred block 304 can be an image of the area corresponding to the volume bar 303 in the social application window 301 after blurring. The size of the blurred block 304 is the same as the size of the background image of the volume bar 303, and the position of the blurred block 304 is the same as the position of the background image of the volume bar 303. For example, the area corresponding to the volume bar 303 in the social application window 301 can be a blank area, and the blurred block 304 will present a solid color blur effect.
[0109] During the volume bar pop-up phase, the volume bar 303, the blur block 304, and the volume adjustment control 305 can be gradually enlarged until the electronic device enters the interface shown in Figure 3c; the interface shown in Figure 3c can be the volume bar stabilization phase.
[0110] The interface shown in Figure 3c can display a social application window 301, a status bar 302, a volume bar 306, a blur block 307, and a volume adjustment control 308; among them, the background image of the volume bar 306, the size and position of the blur block 307 and the volume adjustment control 308 no longer change.
[0111] The size of the interface elements such as volume bar 306, blur block 307, and volume adjustment control 308 during the stable phase of the volume bar is larger than the size of the interface elements such as volume bar 303, blur block 304, and volume adjustment control 305 during the pop-up phase of the volume bar.
[0112] The blurred block 307 can be an image of the area corresponding to the volume bar 306 in the social application window 301 after blurring. The size of the blurred block 307 is the same as the size of the background image of the volume bar 306, and the position of the blurred block 307 is the same as the position of the background image of the volume bar 306. For example, the area corresponding to the volume bar 306 in the social application window 301 may include a user avatar, and the blurred block 307 may include a blurred user avatar.
[0113] The upper edge of the volume bar 306 foreground image can rise or fall accordingly when the user triggers the volume buttons. For example, the volume buttons may include a volume up button and a volume down button; when a trigger operation is received for the volume up button, the electronic device may display the interface shown in Figure 3d, where the upper edge of the volume bar 306 foreground image rises as the user triggers the volume up button. As another example, when a trigger operation is received for the volume down button, the upper edge of the volume bar 306 foreground image falls as the user triggers the volume down button (not shown in the figure).
[0114] The interface shown as d in Figure 3 also corresponds to the volume bar stabilization phase. The volume bar stabilization phase can be understood as the volume bar background image being at a fixed size and no longer changing. Changes in the foreground image of the volume bar 306 will not affect the size of the blur block 307.
[0115] Without any change to the underlying interface, the blurred block 307 in the interface shown in Figure 3d and the blurred block 307 in the interface shown in Figure 3c can present the same effect; for example, in the interface shown in Figure 3d, the area corresponding to the volume bar 306 in the social application window 301 includes the user avatar, and the blurred block 307 can include the blurred user avatar.
[0116] After the volume adjustment is completed, the electronic device can display the interface shown in Figure 3e. The interface shown in Figure 3e can be the volume bar collapse stage; the interface shown in Figure 3e may include a social application window 301, a status bar 302, a volume bar 309, a blur block 310, and a volume adjustment control 311, etc.
[0117] The blurred block 310 can be an image of the area corresponding to the volume bar 309 in the social application window 301 after blurring. The size of the blurred block 310 is the same as the size of the background image of the volume bar 309, and the position of the blurred block 310 is the same as the position of the background image of the volume bar 309. For example, the area corresponding to the volume bar 309 in the social application window 301 can be a blank area, and the blurred block 310 will present a solid color blur effect.
[0118] The volume bar collapse phase is the reverse of the volume bar pop-up phase. During the volume bar exit phase, the volume bar 309, the blurred block 310, and the volume adjustment control 311 can gradually shrink. The size of the interface elements such as the volume bar 309, the blurred block 310, and the volume adjustment control 311 during the volume bar collapse phase is smaller than the size of the interface elements such as the volume bar 306, the blurred block 307, and the volume adjustment control 308 during the volume bar stabilization phase.
[0119] Multiple volume-related interface elements can gradually shrink until they disappear, as shown in the interface shown in Figure 3f. The interface shown in Figure 3f can display the social application window 301 and the status bar 302, but does not display volume-related interface elements.
[0120] It can be understood that a static interface is a lower-level interface that remains unchanged for two or more consecutive frames during the volume bar pop-up phase, the volume bar stabilization phase, and the volume bar collapse phase. For example, in the interfaces shown in Figure 3b to Figure 3e, the lower-level interface is a social application window, and the interface elements in the social application window do not change.
[0121] It should be noted that the interfaces shown in Figure 3a and Figure 3f do not involve real-time blurring. Electronic devices can use HWC to synthesize the interfaces shown in Figure 3a and Figure 3f. The image synthesis process can be referred to the relevant description in Figure 2a.
[0122] The interfaces shown in Figure 3b to Figure 3e correspond to real-time blurred scenes such as the volume bar pop-up stage, the volume bar stabilization stage, and the volume bar collapse stage, respectively. The electronic device can blur the area corresponding to the volume bar in the lower interface to obtain a blurred block. The electronic device stores the blurred block in a stable layer to obtain a blurred layer. The electronic device uses HWC to composite the blurred layer, the lower interface layer, the volume bar layer, and the status bar layer to obtain a real-time blurred interface.
[0123] The electronic device uses HWC compositing during the stages of volume bar not displaying, volume bar popping up, volume bar stabilizing, and volume bar collapsing. The brightness of multiple interfaces, from the interface shown in Figure 3a to the interface shown in Figure 3f, is the same.
[0124] In this way, during the switching between non-real-time blurred scenes and real-time blurred scenes, electronic devices will not force the GPU to perform layer compositing, reducing the switching process between HWC and GPU; when compositing images based on HWC, HWC can use the same color gamut mapping and gamma correction and other image processing technologies to process multiple layers, thereby reducing the bright-dark-bright flickering phenomenon that occurs when the image compositing method is different, and improving the user experience.
[0125] Taking the video display application interface of an electronic device as an example, Figure 4 shows a schematic diagram of the interface of another image synthesis method provided in the embodiment of this application. In the embodiment shown in Figure 4, the lower layer interface can be a dynamic interface; as shown in Figure 4:
[0126] For example, when the screen is on, the electronic device can display the interface shown in Figure 4a; the interface shown in Figure 4a may include: a video application window 401 and a status bar 402. The video application window 401 can display dynamic images, and the status bar 402 can be referred to the relevant description of the status bar 102.
[0127] In the interface shown in Figure 4a, when a trigger operation is received on the volume button, the electronic device can display the interface shown in Figure 4b. The interface shown in Figure 4b can be the volume bar pop-up stage. In the interface shown in Figure 4b, the electronic device can display a video application window 403, a status bar 402, a volume bar 404, a blur block 405, and a volume adjustment control 406, etc. The screen in the video application window 403 is different from the screen in the video application window 401.
[0128] It should be noted that the blur block 405 can be an image of the area in the video application window 403 corresponding to the volume bar 404 after blurring; the size of the blur block 405 is the same as the size of the background image of the volume bar 404, and the position of the blur block 405 is the same as the position of the background image of the volume bar 404. For example, the area in the video application window 401 corresponding to the volume bar 403 may include a person's arm, and the blur block 404 can display the blurred arm.
[0129] During the volume bar pop-up phase, the volume bar 404, the blur block 405, and the volume adjustment control 406 can be gradually enlarged until the electronic device enters the interface shown in Figure 4c; the interface shown in Figure 4c can be the volume bar stabilization phase.
[0130] The interface shown in Figure 4c displays a video application window 407, a status bar 402, a volume bar 406, a blur block 407, and a volume adjustment control 408; the background image of the volume bar 406, the size and position of the blur block 407, and the volume adjustment control 408 remain unchanged. The image in the video application window 407 differs from the image in the video application window 403.
[0131] The size of the interface elements such as the volume bar 408, the blur block 409, and the volume adjustment control 410 during the stable phase of the volume bar is larger than the size of the interface elements such as the volume bar 404, the blur block 405, and the volume adjustment control 406 during the pop-up phase of the volume bar.
[0132] The blurred block 409 can be an image of the area corresponding to the volume bar 408 in the video application window 407 after blurring. The size of the blurred block 409 is the same as the size of the background image of the volume bar 408, and the position of the blurred block 409 is the same as the position of the background image of the volume bar 408. For example, the area corresponding to the volume bar 408 in the video application window 407 may include the user's arm and part of the torso, and the blurred block 404 may include the blurred user's arm and part of the torso.
[0133] In the interface shown in Figure 4c, when a trigger operation is received for the volume up key, the electronic device can display the interface shown in Figure 4d, where the upper edge of the volume bar 408 foreground image rises as the user triggers the volume up key. Alternatively, when a trigger operation is received for the volume down key, the upper edge of the volume bar 408 foreground image falls as the user triggers the volume down key (not shown in the figure).
[0134] The interface shown as d in Figure 4 is in the stable volume bar phase. When the underlying interface changes, the blur block can change accordingly. For example, the interface shown as d in Figure 4 includes a video application window 411 and a blur block 412. The person in the video application window 411 moves forward compared to the person in the video application window 407. When the underlying interface changes, the area in the video application window 411 corresponding to the volume bar 408 may include the user's palm, and the blur block 412 may include a blurred user's palm.
[0135] After the volume adjustment is completed, the electronic device can display the interface shown in Figure 4e. The interface shown in Figure 4e can be the volume bar collapse stage; the interface shown in Figure 4e can include a video application window 413, a status bar 402, a volume bar 414, a blur block 415, and a volume adjustment control 416, etc.; the screen in the video application window 413 is different from the screen in the video application window 411.
[0136] The blurred block 415 can be an image of the area in the video application window 413 corresponding to the volume bar 414 after blurring. The size of the blurred block 415 is the same as the size of the background image of the volume bar 414, and the position of the blurred block 415 is the same as the position of the background image of the volume bar 414. For example, the area in the video application window 413 corresponding to the volume bar 414 can be a blank area, and the blurred block 415 will present a solid color blur effect.
[0137] The volume bar collapse phase is the reverse of the volume bar pop-up phase. During the volume bar exit phase, the volume bar 414, the blurred block 415, and the volume adjustment control 416 can gradually shrink. The size of the interface elements such as the volume bar 414, the blurred block 415, and the volume adjustment control 416 during the volume bar collapse phase is smaller than the size of the interface elements such as the volume bar 408, the blurred block 412, and the volume adjustment control 410 during the volume bar stabilization phase.
[0138] Multiple volume-related interface elements can gradually shrink until they disappear, as shown in the interface in Figure 4f. The interface shown in Figure 4f can display the video application window 417 and the status bar 402, but does not display volume-related interface elements; the screen in the video application window 417 is different from the screen in the video application window 413.
[0139] The interfaces shown in Figure 4a and Figure 4f do not require real-time blurring. The electronic device can use HWC to composite the interfaces shown in Figure 4a and Figure 4f. The interfaces shown in Figure 4b to Figure 4e correspond to real-time blurring scenarios such as the volume bar pop-up stage, the volume bar stabilization stage, and the volume bar collapse stage, respectively. The electronic device can blur the area corresponding to the volume bar in the lower interface to obtain a blurred block; the electronic device stores the blurred block in a stable layer to obtain a blurred layer; the electronic device uses HWC to composite the blurred layer, the lower interface layer, the volume bar layer, and the status bar layer to obtain the corresponding real-time blurred interface.
[0140] The electronic device uses HWC compositing during the stages of volume bar not displaying, volume bar popping up, volume bar stabilizing, and volume bar collapsing. The brightness of multiple interfaces, from the interface shown in Figure 4a to the interface shown in Figure 4f, is the same.
[0141] In this way, during the switching between non-real-time blurred scenes and real-time blurred scenes, electronic devices will not force the GPU to perform layer compositing, reducing the switching process between HWC and GPU; when compositing images based on HWC, HWC can use the same color gamut mapping and gamma correction and other image processing technologies to process multiple layers, thereby reducing the bright-dark-bright flickering phenomenon that occurs when the image compositing method is different, and improving the user experience.
[0142] It should be noted that this application embodiment only uses the volume adjustment scenario as an example to illustrate the interface of the image synthesis method in this application embodiment. Figure 5 shows another part of the real-time blurred scene, see Figure 5:
[0143] The interface shown in Figure 5a is a message pop-up scenario: when an electronic device receives a message notification, it can display a message pop-up. For example, if an electronic device receives a message notification from a fitness app, it will display message pop-up 501 on the top layer of the desktop. Message pop-up 501 may include step count and calorie consumption prompts, etc.
[0144] In the message pop-up scenario, the lower-level interface is the desktop and desktop wallpaper. The blurred block can be, for example, an image of the desktop and desktop wallpaper that has been blurred to correspond to the area of the message pop-up. The size and position of the blurred block are the same as the size and position of the message pop-up 501. For example, the blurred block includes blurred portion of time information.
[0145] The interface shown in Figure 5b is a control center scenario: when an electronic device receives a swipe-down operation from the top of the screen (such as the top right), the control center interface can be displayed; the control center interface may include one or more shortcut controls, such as WLAN controls and Bluetooth controls.
[0146] In the control center scenario, the lower-level interface consists of the desktop and desktop wallpaper. The blurred block can be, for example, an image of the corresponding area of the desktop and desktop wallpaper that has been blurred to fit the control center interface; the size of the blurred block can be full screen. For example, the blurred block includes blurred application icons and time information.
[0147] The interface shown in Figure 5c is a pull-down notification scenario: When an electronic device receives a swipe-down operation from the top of the screen (such as the top left), it can display a pull-down notification interface; the pull-down notification interface may include time information, one or more message notifications, and a delete control, etc.
[0148] In a pull-down notification scenario, the lower-level interface is the desktop and desktop wallpaper. The blurred block can be, for example, an image of the area on the desktop and desktop wallpaper that corresponds to the pull-down notification interface, blurred; the size of the blurred block can be full screen. For example, the blurred block includes blurred application icons and time information.
[0149] The interface shown in Figure 5d is a restart or shutdown scenario: when a long press operation is received on the power button, the electronic device can display a restart or shutdown interface; the restart or shutdown interface may include restart controls and shutdown controls, etc.
[0150] In a restart or shutdown scenario, the lower-level interface consists of the desktop and desktop wallpaper. The blurred block can be, for example, an image of the area on the desktop and desktop wallpaper corresponding to the restart or shutdown interface that has been blurred; the size of the blurred block can be full screen. For example, the blurred block includes blurred application icons and time information.
[0151] The interface shown in Figure 5e is a multitasking scenario: when a swipe-up gesture is received from the bottom of the screen and paused, the electronic device can display a multitasking interface; the multitasking interface may include social application cards and delete controls, etc.; social application cards can be used to indicate that social applications are running in the background of the electronic device.
[0152] In a multitasking scenario, the lower-level interface consists of the desktop and desktop wallpaper. The blurred block can be, for example, an image of the corresponding area of the desktop and desktop wallpaper that has been blurred. The size of the blurred block can be full screen. For example, the blurred block includes blurred application icons and time information.
[0153] The interface shown in Figure 5f is a sidebar scenario: when a trigger operation is received by dragging inward from the left or right edge of the screen, the electronic device can display the sidebar 502; the sidebar 502 includes multiple shortcut controls, such as a favorites control, a note application control, and a gallery application control.
[0154] In the sidebar scenario, the lower layer interface consists of the desktop and desktop wallpaper. The blurred block can be, for example, an image of the desktop and desktop wallpaper that has been blurred to correspond to the area of the sidebar 502; the size of the blurred block is the same as the size of the sidebar 502. For example, the blurred block includes blurred portions of application icons and time information.
[0155] The embodiments of this application will not list all real-time blurred scenes.
[0156] To better understand the embodiments of this application, the structure of the electronic device of this application is described below:
[0157] Figure 6 shows a schematic diagram of the structure of electronic device 100. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an embedded secure element (eSE) 196, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0158] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0159] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.
[0160] Electronic device 100 implements display functions through GPU, display screen 194, and application processor.
[0161] The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. The GPU can be used for image overlay and blurring. In this embodiment, the electronic device 100 can generate a blur layer via the GPU; the blur layer can be applied to the compositing process of copied frame images, or reused in the compositing process of skipped frame images.
[0162] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In this embodiment, the display screen 194 can be used to display the user interface shown in Figures 3-5.
[0163] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.
[0164] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.
[0165] In this embodiment, the touch sensor 180K can be used to receive operations for triggering real-time blurred scenes; such as operations for triggering volume adjustment scenes, operations for triggering message pop-up scenes, operations for triggering control center scenes, operations for triggering pull-down notification scenes, operations for triggering restart or shutdown scenes, operations for triggering multitasking scenes, or operations for triggering sidebar scenes, etc.
[0166] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.
[0167] Figure 7 is a software structure block diagram of an electronic device according to an embodiment of this application.
[0168] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer (or framework layer), the Android runtime and system libraries, the hardware abstraction layer, and the kernel layer. For ease of illustration, the software architecture diagram of the electronic device shown in Figure 7 also includes the hardware structures shown in Figure 6, such as the graphics processing unit (GPU) and the display screen.
[0169] The application layer can include a series of application packages. As shown in Figure 7, application packages can include social applications and video applications, etc.
[0170] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 7, the application framework layer may include a window manager, content provider, view system, resource manager, notification manager, etc.
[0171] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.
[0172] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.
[0173] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A user interface can consist of one or more views. For example, a user interface that includes a desktop application icon can include views for displaying text and views for displaying images.
[0174] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.
[0175] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.
[0176] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java needs to call, and the other part is the core Android library itself. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.
[0177] The system library can include multiple functional modules. For example: a layer management module (surface flinger), a capture display interface, a 2D graphics engine (e.g., SGL), a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGLES), etc.
[0178] The layer management module is used to control image compositing. Specifically, the layer management module can receive graphics data buffers from multiple sources, composite them, and then send them to display hardware (e.g., a display screen). In this embodiment, the layer management module can be used to create a blur buffer to store blur layers; the layer management module can also instruct the GPU to composite frame images or the HWC to composite frame images.
[0179] The screenshot blur interface can be used to call the GPU to generate a blur layer and draw the blur layer onto the blur buffer. A stable blur layer can be used for compositing the current frame image or reused in the compositing process of subsequent frames.
[0180] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0181] A 2D graphics engine is a graphics engine for 2D drawing.
[0182] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.
[0183] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.
[0184] The Hardware Abstraction Layer (HAL) can contain multiple library modules, such as a hardware composer (HWC) and a camera library. The Android system can load the corresponding library modules for the device hardware, thereby enabling the application framework layer to access the device hardware.
[0185] The Hardware Hybrid Renderer (HWC) primarily provides hardware support for image compositing systems, supporting layer compositing and display modules. In this embodiment, HWC can be used to composite layers such as lower-level interfaces, blur blocks, and volume bars in real-time blurred scenes, avoiding forced GPU-based frame image compositing.
[0186] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.
[0187] The hardware layer may include hardware such as displays and GPUs.
[0188] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.
[0189] Figure 8 shows a flowchart of an image synthesis method provided in an embodiment of this application, as shown in Figure 8:
[0190] S801, In response to the first operation, the electronic device acquires multiple layers of the first frame image; the multiple layers include a lower interface layer and a target layer; the target layer may carry blur parameters; the blur parameters include the number of blur layers, the name of the blur layer, the size of the blur layer, the position of the blur layer, and the blur radius, etc.
[0191] The first operation can be an operation used to trigger a real-time blurred scene. For example, the first operation may be: receiving a trigger operation on the volume button in the interface shown in Figure 3a or Figure 4a; receiving a message notification in the interface shown in Figure 5a; receiving a swipe-down operation from the top of the screen (e.g., top right) in the interface shown in Figure 5b; receiving a swipe-down operation from the top of the screen (e.g., top left) in the interface shown in Figure 5c; a long press operation on the power button in the interface shown in Figure 5d; or a swipe-up and pause operation from the bottom of the screen in the interface shown in Figure 5e, etc. This application embodiment does not limit the operation used to trigger a real-time blurred scene. For example, the electronic device may also enter a real-time blurred scene based on a voice command.
[0192] The first frame image can be the current frame image to be displayed. Multiple layers include a lower interface layer and a target layer. The target layer can carry blur parameters and can be, for example, a volume bar layer, a control center interface layer, a drop-down notification interface layer, a restart or shutdown interface layer, or a multitasking interface layer. The lower interface layer can be the layer below the target layer. Examples of lower interface layers include: the social application window layer in Figure 3, the video application window layer in Figure 4, and the desktop and desktop wallpaper layers in Figure 5.
[0193] The blur parameter can be used to instruct the electronic device to blur the area in the lower interface corresponding to the target layer, resulting in a blurred layer; the blurred layer can be, for example, the layer corresponding to the blurred block in the interface shown in Figures 3-5. Optionally, multiple layers may also include other layers, which can be layers that do not overlap with the blurred layer.
[0194] For example, after receiving the first operation, the current frame image to be displayed is the interface shown in Figure 3b. The electronic device can obtain multiple layers and layer parameters used to draw this interface; the multiple layers include, for example, the social application window layer 301, the status bar layer 302, and the volume bar layer 303; the layer parameters can be used to represent the positional and hierarchical relationships of the multiple layers. Among them, the lower interface layer can be the social application window layer below the blurred layer; the target layer can be, for example, the volume bar layer. The status bar layer and the blurred layer have no positional overlap, and other layers can be, for example, the status bar layer.
[0195] The blur layer is generated in real time based on the social application window layer and the volume bar layer. That is, in step S801, the electronic device can obtain the lower interface layer, the target layer, and other layers, but the electronic device does not obtain the blur layer.
[0196] The number of blur layers can be one or more. For example, in the volume adjustment scene, the interface shown in Figure 3b includes a blur block; in the message notification scene, the interface shown in Figure 5a includes a blur block. Furthermore, when multiple real-time blur scenes are superimposed, the number of blur layers can be multiple; for example, in the volume adjustment and message notification scenes, the number of blur layers can be two.
[0197] The name of a blurred layer can be used to identify it. For example, when there are multiple blurred layers, electronic devices can distinguish between them by their names.
[0198] The size of the blur layer can be used to identify its shape and size, while its position can be used to identify its location above the underlying interface layer. The blur radius can be used to define the blur intensity. Specifically, the size of the blur layer is the same as the size of the target layer, and the position of the blur layer is the same as the target layer.
[0199] S802, Electronic device creates fuzzy buffer.
[0200] A blur buffer can be a pre-created buffer that caches blur blocks when blur parameters are present in the current frame image. For example, an electronic device can create a blur buffer using a surface flinger after detecting blur parameters.
[0201] The blur buffer can exist stably and can be a memory area that can exist and be reused continuously in a real-time blurred scene. After the current frame image is synthesized, the electronic device will not destroy the blur buffer. That is, the life cycle of the blur buffer is the same as the life cycle of the target layer (e.g., the volume bar layer).
[0202] It should be noted that step S802 in this embodiment is optional. For example, in a real-time blurred scene, before the first generation of the blurred layer, the electronic device may execute step S802 to create a blurred buffer to store the blurred layer. When generating blurred layers subsequently, since the blurred buffer can survive the entire real-time blurred scene process, the electronic device can directly use the blurred buffer without executing step S802, so as to avoid repeatedly creating the blurred buffer and wasting memory resources.
[0203] S803 and GPU perform blur processing on the lower interface layer according to the blur parameters to obtain a blurred layer; the blurred layer is cached in the blur buffer.
[0204] GPUs support layer overlay and layer blurring. In this embodiment, the electronic device can invoke the GPU's blurring capabilities through a screenshot blurring interface to obtain a blurred layer.
[0205] Specifically, the process of the GPU generating the blur layer can be illustrated in Figure 9:
[0206] The S901 GPU can obtain the blur buffer, the lower interface layer, and blur parameters.
[0207] In this embodiment, the lower interface layer can be one or more layers with visible elements below the blurred layer. For example, the lower interface layer can be an opaque layer below the blurred layer; for instance, in the interface shown in Figure 3b, the layer below the blurred block 304 may include a social application window layer, which is an opaque layer, so the lower interface layer can be the social application window layer. The lower interface layer can also be multiple layers with specific transparency levels and opaque layers below the blurred layer; for example, in the interface shown in Figure 5a, the layer below the blurred block may include a desktop layer and a desktop wallpaper layer, where the desktop layer is a transparent layer and the desktop wallpaper layer is an opaque layer, so the lower interface layer may include the desktop layer and the desktop wallpaper layer. This embodiment does not limit the number or form of the lower interface layers.
[0208] It should be noted that, in this embodiment of the application, the electronic device can trigger the GPU's blurring process by calling the screenshot blurring interface. In step S901, the electronic device can call the screenshot blurring interface and input the blur buffer, the lower interface layer, and the blur parameters as input parameters to the screenshot blurring interface. Subsequently, the screenshot blurring interface can call the GPU to execute, for example, steps S902-S905.
[0209] S902, the GPU draws the lower-level interface layer onto the blur buffer.
[0210] When the lower-level interface layer is a single layer, it is drawn onto a blank blur buffer. When there are multiple lower-level interface layers, they are drawn sequentially onto the blank blur buffer according to their hierarchy. For example, when the lower-level interface layer is a social application window layer, the GPU draws the social application window layer onto the blur buffer. When the lower-level interface layer includes a desktop layer and a desktop wallpaper layer, the GPU draws the desktop wallpaper layer onto the blur buffer according to the hierarchy; the GPU also draws the desktop layer on top of the blur buffer including the desktop wallpaper layer according to the hierarchy, and aligns the desktop wallpaper layer and the desktop layer according to their position. The electronic device receives a blur buffer that includes the lower-level interface layers.
[0211] The S903 and GPU perform blurring on the lower interface layer based on the blur radius to obtain the blurred lower interface layer.
[0212] The blurring process can refer to Gaussian blurring or Kawase blurring, etc., and this application does not specifically limit the blurring process method. The GPU can determine the blur intensity based on the blur radius; whereby the blur radius and blur intensity are positively correlated.
[0213] The lower-level interface layer may include one or more layers. When the lower-level interface layer includes one layer, the GPU can blur the single layer; when the lower-level interface layer includes multiple layers, the GPU needs to blur multiple layers, resulting in higher GPU power consumption and load. In this embodiment, the GPU can take screenshots of multiple lower-level interface layers, and the screenshots may include visible elements from multiple lower-level interface layers; the blurring effect of the lower-level interface layer screenshots is the same as the effect of blurring multiple layers separately. This reduces the number of blurring operations, thereby reducing GPU power consumption and load.
[0214] Electronic devices can obtain a blur buffer that includes the blurred lower interface layer.
[0215] S904 and GPU use the size and position of the blurred layer to cut out the underlying interface layer after blurring, thus obtaining the blurred layer.
[0216] The GPU can perform image cutout on the blurred lower interface layer, extracting a blurred layer of the same size and position as the target layer from the blurred lower interface layer. For example, the image corresponding to the volume bar layer can be extracted from the blurred lower interface layer to obtain a blurred layer. This application does not limit the specific method of image cutout processing.
[0217] It should be noted that this embodiment of the application uses the example of first blurring the lower interface layer and then cutting out the blurred lower interface layer to illustrate the GPU blurring process in this embodiment. Electronic devices may also first cut out the lower interface layer or screenshot, and then blur the cut-out lower interface layer or screenshot. This embodiment of the application does not limit the order of steps S903 and S904.
[0218] S905 GPU clears the blurred lower-level interface layer in the blur buffer and draws the blurred layer to the blur buffer.
[0219] Understandably, the size of the blurred layer is the same as the size of the target layer (e.g., the volume bar layer); in the blurred lower interface layer, except for the cutout area, other blurred areas do not need to be rendered into the user interface. The electronic device can clear the blurred lower interface layer and draw the blurred layer in the blur buffer, thus obtaining a blur buffer that includes the blurred layer.
[0220] It should be noted that although the electronic device calls the GPU for blurring in steps S901-S905, the process shown in steps S901-S905 does not belong to the GPU compositing process.
[0221] The differences between the GPU blurring process in the embodiment shown in Figure 9 and the GPU compositing process in the embodiment shown in Figure 2 will be explained below.
[0222] 1) The triggering methods for GPU blurring are different:
[0223] In the embodiment shown in Figure 9, the electronic device triggers the GPU blurring process by calling the screenshot blurring interface. The process of calling the screenshot blurring interface can be in the commit phase; the commit phase can be used to collect parameters required for image compositing, such as obtaining multiple layers, blur layers, layer parameters, and compositing identifiers. The compositing identifier can be used to identify the compositing method, such as indicating whether GPU compositing or HWC compositing is used. In this embodiment, the compositing identifier can be used to indicate that HWC is used for compositing. The electronic device can input the parameters obtained in the commit phase into HWC, and the electronic device enters the composite phase; in the composite phase, the electronic device can use HWC for compositing. That is, in the embodiment shown in Figure 9, the GPU blurring process precedes the image compositing process.
[0224] In the embodiment shown in Figure 2, the electronic device triggers a GPU blurring process through GPU compositing. The GPU compositing process can correspond to the composite stage. The GPU compositing process may include a GPU blurring process and a GPU overlay process. After obtaining parameters such as a desktop layer, desktop wallpaper layer, and volume bar layer, a compositing identifier is determined to indicate the use of GPU compositing. The electronic device determines to execute the GPU compositing process, and the GPU performs GPU blurring and GPU overlay processing. That is, in the embodiment shown in Figure 2, the GPU blurring process occurs during the image compositing process.
[0225] 2) Different buffers:
[0226] In the embodiment shown in Figure 9, the electronic device uses a buffer created by the surface flinger as a blur buffer and inputs the blur buffer as an input parameter into the screenshot blur interface. The blur buffer can be used to store blur layers.
[0227] In the embodiment shown in Figure 2, after determining to perform GPU compositing based on the compositing identifier, the GPU can request a buffer as a GPU buffer to store intermediate calculation results in GPU compositing or to store the composite image. For example, the GPU buffer can be used to store images that are overlaid, such as desktop wallpaper layers, desktop layers, blur layers, and volume bar layers.
[0228] 3) Different blur layers:
[0229] In the embodiment shown in Figure 9, the electronic device creates a blur buffer through a surface flinger. After the blur layer is cached in the blur buffer, the blur layer in the blur buffer can exist stably. In addition, the blur layer in the blur buffer has a long lifespan, and the blur layer in the blur buffer can be applied to the process of multi-frame image synthesis (the process of reusing blur layers can be explained later with reference to Figure 10, which will not be elaborated here).
[0230] In the embodiment shown in Figure 2, the electronic device requests a GPU buffer via the GPU. During GPU compositing, the GPU can sequentially draw, for example, a desktop wallpaper layer and a desktop layer into the GPU buffer for overlay. The GPU copies the overlay result of the desktop wallpaper layer and the desktop layer to a temporary buffer, where it performs blurring. After obtaining the blurred layer, the GPU draws it onto the GPU buffer; then, the GPU draws the volume bar layer onto the GPU buffer, resulting in the overlaid image. The blurred layer is temporarily stored in the temporary buffer, which serves only as a transition during the GPU compositing process. After drawing the blurred layer onto the GPU buffer, the GPU clears the temporary buffer. In GPU compositing, the temporary buffer storing the blurred layer cannot exist stably. Furthermore, the lifespan of the overlaid image in the GPU buffer is short; the blurred layer in the GPU buffer can only be applied to the compositing of a single frame.
[0231] Therefore, in the embodiment shown in Figure 9, although the electronic device calls the GPU to blur the lower interface layer, the process shown in Figure 9 is not part of GPU compositing and will not trigger GPU compositing.
[0232] In this way, the GPU can generate a stable blur layer, which is convenient for subsequent HWC compositing in the image compositing process and for reusing the blur layer when compositing adjacent frames.
[0233] S804 and HWC combine the blurred layer, the lower interface layer, and the target layer to obtain the first frame image.
[0234] In this embodiment, after obtaining the blurred layer, the electronic device does not need to perform further blurring on the layer during image compositing; the electronic device does not need to force the use of GPU compositing. In this case, the electronic device can use HWC to compose the first frame image.
[0235] For example, HWC can obtain a blur layer, a lower interface layer, and a target layer. In the hierarchy, the blur layer can be located between the lower interface layer and the target layer. Taking the interface shown in Figure 3b as an example, the lower interface layer can be the layer of the social application window 301, the target layer can be the layer of the volume bar 303, and the blur layer can be the layer of the blur block 304. Among these, in the hierarchy of the current frame image, the volume bar 303 is located above the blur block 304, and the blur block 304 is located above the social application window 301.
[0236] During the HWC compositing process, HWC draws the lower interface layer (e.g., a social application window layer) onto the HWC buffer; HWC draws the blur layer onto the HWC buffer containing the lower interface layer; HWC draws the target layer (e.g., a volume bar layer) onto the HWC buffer containing both the lower interface layer and the blur layer, thus obtaining the first frame image; the electronic device can display the first frame image as shown in Figure 3b.
[0237] Optionally, the multiple layers may also include other layers, such as the status bar layer 302 in the interface shown in Figure 3b. S804 can also be: HWC composites the blurred layer, the lower interface layer, the target layer, and other layers to obtain the first frame image. This application embodiment does not limit this.
[0238] In this embodiment, when a real-time blurred scene is detected, the electronic device can pre-generate a blurred layer, eliminating the need to generate a blurred layer during image compositing and thus avoiding the use of the GPU for image compositing. The electronic device can treat the blurred layer as a regular layer and pass it to HWC for image compositing along with the underlying interface layer, target layer, and / or other layers.
[0239] It should be noted that, taking the volume adjustment scene in Figure 3 as an example, the social application window 301 and status bar 302 in the interface shown in Figure 3a are composited using HWC. In the interface shown in Figure 3b, the social application window 301, status bar 302, volume bar 303, blur layer 304, and volume adjustment control 305 are composited using HWC; among them, the blur layer 304 is obtained through GPU blurring. In the interface shown in Figure 3b, the social application window 301, status bar 302, volume bar 303, and volume adjustment control 305 do not involve GPU processing; the electronic device uses the same color gamut mapping and gamma correction image processing techniques employed by HWC to process the non-blurred layers in both the interface shown in Figure 3a and the interface shown in Figure 3b. Therefore, when switching from the interface shown in Figure 3a to the interface shown in Figure 3b, the overall brightness of the interface remains unchanged.
[0240] In this way, the previous frames can be images of non-real-time blurred scenes, and the electronic device uses HWC compositing; the current frame can be an image of a real-time blurred scene, and the electronic device can still use HWC compositing, reducing the switching process between HWC compositing and GPU compositing; it can reduce the power consumption, load and rendering latency of the electronic device when forced to use GPU compositing in real-time blurred scenes; at the same time, it reduces the flickering problem of bright-dark-bright caused by switching compositing methods.
[0241] In this embodiment, the blur layer can exist stably. When processing multiple adjacent frames, the electronic device can reuse the blur layer to reduce the GPU blurring process, thereby further reducing the power consumption and load of the GPU. The process of reusing the blur layer in this embodiment is described below with reference to Figure 10, as shown in Figure 10:
[0242] Taking the current frame image as the second frame image, and the second frame image and the first frame image as adjacent frame images as an example.
[0243] S1001, The electronic device acquires multiple layers of the second frame image and the blurred layer of the first frame image; the multiple layers include a lower interface layer and a target layer, the lower interface layer is located below the target layer, and the target layer carries blur parameters.
[0244] The second frame image can be the next frame image after the first frame image. The name of the lower interface layer of the first frame image is the same as the name of the lower interface layer of the second frame image; the name of the target layer of the first frame image is the same as the name of the target layer of the second frame image; for example, the first frame image is the interface shown in Figure 3c, and the second frame image is the interface shown in Figure 3d. The names of the lower interface layers of the first and second frames image are both used to indicate the social application window layer, and the names of the target layers of the first and second frames image are both used to indicate the volume bar layer.
[0245] Optionally, the second frame image may also include other layers among its multiple layers; these other layers can be layers that do not overlap with the blurred layer. For example, in the interface shown in Figure 3d, the other layer is the status bar layer.
[0246] It should be noted that when the blurred layer of the first frame image is different from the blurred layer of the second frame image, using the blurred layer of the previous frame image may lead to problems such as content mismatch between the blurred layer and the lower layer or size and position mismatch between the blurred layer and the upper layer. Therefore, in this embodiment, the electronic device can reuse the blurred layer of the previous frame image when it is determined that the blurred layers of two adjacent frames are the same or similar; when the blurred layers of two adjacent frames are different, the blurred layer of the current frame image is updated. The conditions for reusing the blurred layer will be described later in this embodiment with reference to Figures 11-13, and will not be repeated here.
[0247] The concept of similar blur layers can be understood as follows: when the screen refresh rate is high, the difference between the previous frame image and the current frame image is small. There may be slight differences between the blur layer of the previous frame image and the blur layer of the current frame image, but in the scenario of high screen refresh rate, the slight difference between the two can be ignored. The electronic device still uses the blur layer of the previous frame image as the blur layer of the current frame image, thereby realizing the reuse of blur layers.
[0248] S1002 and HWC composite the blurred layer, the lower interface layer, and the target layer to obtain the second frame image.
[0249] In this embodiment, after reusing the blurred layer, the electronic device does not need to perform further blurring on the layer during image compositing; the electronic device does not need to force the use of GPU compositing. In this case, the electronic device can use HWC to compose the second frame image.
[0250] For example, HWC can obtain a blur layer, a lower interface layer, and a target layer. In the hierarchy, the blur layer can be located between the lower interface layer and the target layer. Taking the interface shown in Figure 3d as an example, the lower interface layer can be the social application window 301 layer, the target layer can be the volume bar 306 layer, and the blur layer can be the blur block 307 layer. Among these, in the hierarchy of the current frame image, the volume bar 306 is located above the blur block 307, and the blur block 307 is located above the social application window 301.
[0251] During the HWC compositing process, HWC draws the lower interface layer (e.g., a social application window layer) onto the HWC buffer; HWC draws the blur layer onto the HWC buffer containing the lower interface layer; HWC draws the target layer (e.g., a volume bar layer) onto the HWC buffer containing both the lower interface layer and the blur layer, resulting in the second frame image; the electronic device can then display the second frame image as shown in Figure 3d.
[0252] Optionally, the multiple layers may include other layers, such as the status bar 302 layer in the interface shown in Figure 3d. S1002 may also be: HWC composites the blurred layer, the lower interface layer, the target layer, and other layers to obtain the second frame image. This application embodiment does not limit this.
[0253] In this way, when the blur layers of two adjacent frames are the same or similar, the electronic device does not need to perform GPU blur processing. The electronic device uses the blur layer of the previous frame as the blur layer of the current frame, reducing the number of GPU blur processing operations by reusing the blur layer, thereby reducing the power consumption and load of the GPU. In addition, the electronic device uses HWC to composite multiple layers in the second frame with the blur layer, which can reduce the process of forcing the GPU to composite in real-time blurred scenes, thereby reducing the bright-dark-bright flickering problem caused by switching compositing methods.
[0254] The following description, in conjunction with Figures 11-13, explains the two determination conditions for reusing the blurred layer in the embodiments of this application.
[0255] Taking the volume adjustment scenario as an example, Figure 11 shows a schematic diagram of the changes in the blur layer in the volume adjustment scenario of this application embodiment, as shown in Figure 11:
[0256] In the same frame, the size of the blurred layer is the same as the size of the target layer. Taking the volume bar layer as the target layer as an example, the changes in the blurred layer are explained using 15 frames of images from the appearance to the disappearance of the volume bar.
[0257] The volume bar's appearance and disappearance can be divided into three stages: the volume bar pop-up stage, the volume bar stabilization stage, and the volume bar exit stage. During this process, the blurred layer of the first frame image can be the first frame blur layer; the blurred layer of the second frame image can be the second frame blur layer, and so on, following the chronological order. Figure 11 shows the changes in the blurred layers across 15 frames.
[0258] It should be noted that the blur parameters of the blur layer may also include variation parameters of the blur layer; for example, the variation parameters may include a first value and a second value. The first value indicates that the size and position of the blur layer in the current frame image are different from those in the previous frame image, and the second value indicates that the size and position of the blur layer in the current frame image are the same as those in the previous frame image. The first value may be, for example, True, and the second value may be, for example, Flat; the first value may be, for example, 1, and the second value may be, for example, 0; or the first value may be, for example, 0, and the second value may be, for example, 1. This application does not limit the form and data of the variation parameters.
[0259] The first frame image can be the interface displayed by the electronic device when a trigger operation is received for the volume button, as shown in Figure 3a; the blur layer of the first frame can be empty, and the blur layer of the first frame does not include the changing parameters.
[0260] The blur layers from frame 2 to frame 5 correspond to the volume bar pop-up phase, as shown in Figure 3b. During the volume bar pop-up phase, the size of the blur layers gradually increases; for example, the size of the blur layer in frame 2 is smaller than that in frame 3; the size of the blur layer in frame 3 is smaller than that in frame 4; and the size of the blur layer in frame 4 is smaller than that in frame 5. During the volume bar pop-up phase, the parameters for the changes in the blur layers from frame 2 to frame 5 are all at the first value.
[0261] From the 6th frame blur layer to the 10th frame blur layer, the volume bar stabilizes, and the size of the blur layer remains unchanged, as shown in the interfaces in Figure 3c and Figure 3d. For example, if the size of the 6th frame blur layer is the same as the 5th frame blur layer, then the change parameter of the 6th frame blur layer is the second value; similarly, the change parameter of the 7th frame blur layer is the same as the 6th frame blur layer, and the change parameter of the 7th frame blur layer is also the second value. Likewise, the change parameters of the 8th to 10th frame blur layers are all the second value.
[0262] The blur layer from frame 11 to frame 14 corresponds to the volume bar collapsing phase, as shown in the interface in Figure 3e. During the volume bar collapsing phase, the blur layer gradually decreases in size until it disappears. For example, the size of the blur layer in frame 11 is larger than that in frame 12; the size of the blur layer in frame 12 is larger than that in frame 13; and the size of the blur layer in frame 13 is larger than that in frame 14. During the volume bar collapsing phase, the change parameters of the blur layer from frame 11 to frame 14 are all at the first value.
[0263] The 15th frame image can be the interface after the volume adjustment is completed; as shown in f in Figure 3; the blur layer of the 15th frame can be empty, and the blur layer of the 15th frame does not include the changed parameters.
[0264] It should be noted that, in this application embodiment, the blur layer can be divided into a blur layer of a copied frame image and a blur layer of a skipped frame image. The copied frame image can be the current frame image used to update the blur layer, and the skipped frame image can be a frame image that uses the blur layer of the previous copied frame image as the blur layer of the current frame image. For example, when displaying a copied frame image, the electronic device can call the GPU to generate the blur layer, as shown in the embodiment of Figure 8; when displaying a skipped frame image, the electronic device can reuse the blur layer corresponding to the previous copied frame image, as shown in the embodiment of Figure 10.
[0265] In this embodiment, when the parameter is a first value, the size of the blurred layer changes; the electronic device can adjust the ratio of the skipped frame image to the copied frame image according to the screen refresh rate. When the parameter is a second value, the size of the blurred layer does not change; the electronic device can adjust the ratio of the skipped frame image to the copied frame image according to the changes in the underlying interface layer.
[0266] Figure 12 shows a schematic diagram of a process for adjusting the ratio of skipped frame images and copied frame images by screen refresh rate according to an embodiment of this application. As shown in Figure 12:
[0267] Electronic devices can refresh frame images according to the screen refresh rate. During the volume bar pop-up and volume bar collapse phases, the screen refresh rate is the same as the refresh rate of the underlying interface; for example, if the underlying interface is a social application window, the screen refresh rate may be related to the refresh rate of the social application window it sends.
[0268] Understandably, during the volume bar pop-up phase, the electronic device can go through a process of not displaying the volume bar, gradually enlarging the volume bar, and reaching its maximum size (e.g., the same size as the volume bar during the stable phase). With the duration of the volume bar pop-up phase and the maximum size of the volume bar remaining constant, the more frame images there are, the smaller the single change in volume. Similarly, with the duration of the volume bar collapse phase and the maximum size of the volume bar remaining constant, the more frame images there are, the smaller the single change in volume. In other words, the higher the screen refresh rate, the smaller the change in the blur layer.
[0269] Figure 12a corresponds to the process of adjusting the ratio of copied frame images to skipped frame images at a high screen refresh rate, such as 120Hz. Figure 12b corresponds to the process of adjusting the ratio of copied frame images to skipped frame images at a low screen refresh rate, such as 60Hz or 90Hz.
[0270] For example, at a high screen refresh rate, an electronic device can refresh 11 frames of images within time t; such as frame 1 to frame 11. At a low screen refresh rate, an electronic device can refresh 6 frames of images within time t; such as frame 1' to frame 6'.
[0271] The comparison shows that the higher the screen refresh rate, the smaller the change in the blur layer between adjacent frames. For example, at a high screen refresh rate, the change in the size of the blur layer between the first and second frames is smaller than the change in the size of the blur layer between the first and second frames at a low refresh rate. At a high screen refresh rate, the blur layer of the next frame can reuse the blur layer of the previous frame; the size change of the blur layer is small and not easily noticed by the user. At a low screen refresh rate, if the blur layer of the previous frame is used as the blur layer of the next frame, the size change of the blur layer is larger, which may cause stuttering in the animation during the volume pop-up phase.
[0272] In this embodiment, when the screen refresh rate is greater than a first refresh rate threshold, the electronic device can use a copy frame image and a skip frame image during the volume bar pop-up or volume bar collapse phase. For example, the first refresh rate threshold is 90Hz; at 120Hz, the display order of the copy frame image and the skip frame image in the electronic device can be: one copy frame image followed by one skip frame image. The first frame image is a copy frame image, the second frame image is a skip frame image, the third frame image is a copy frame image, the fourth frame image is a skip frame image, and so on, as shown in Figure 12a.
[0273] When the screen refresh rate is less than or equal to a first refresh rate threshold, the electronic device can use a copied frame image during the volume bar pop-up or volume bar collapse phases, and does not use a skipped frame image. For example, if the first refresh rate threshold is 90Hz, at 60Hz, the display order of the copied frame image and the skipped frame image in the electronic device can be: every frame is a copied frame image. The first frame is a copied frame image, the second frame is a copied frame image, the third frame is a copied frame image, the fourth frame is a copied frame image, and so on, as shown in Figure 12b.
[0274] Optionally, in this embodiment, the electronic device can adjust the ratio of the copied frame image to the skipped frame image according to the screen refresh rate. For example, Figure 12a shows the case where the ratio of the skipped frame image to the copied frame image is equal to 1; in other embodiments, the ratio of the skipped frame image to the copied frame image may be greater than 1.
[0275] For example, when the screen refresh rate is greater than the second refresh rate threshold, the ratio of skipped frame images to copied frame images is 2, and the second refresh rate threshold is higher than the first refresh rate threshold. The display order of copied frame images and skipped frame images in an electronic device can be: one copied frame image followed by two skipped frame images. For example, the first frame is a copied frame image, the second frame is a skipped frame image, the third frame is a skipped frame image, the fourth frame is a copied frame image, the fifth frame is a skipped frame image, the sixth frame is a skipped frame image, and so on.
[0276] For example, when the screen refresh rate is greater than the third refresh rate threshold, the ratio of skipped frame images to copied frame images is 3, and the third refresh rate threshold is higher than the second refresh rate threshold. The display order of copied frame images and skipped frame images in an electronic device can be: one copied frame image followed by three skipped frame images. For example, the first frame is a copied frame image, the second frame is a skipped frame image, the third frame is a skipped frame image, the fourth frame is a skipped frame image, the fifth frame is a copied frame image, the sixth frame is a skipped frame image, the seventh frame is a skipped frame image, the eighth frame is a skipped frame image, and so on.
[0277] The higher the screen refresh rate, the higher the ratio of skipped frame images to copied frame images. In this embodiment, the ratio of skipped frame images to copied frame images can be M, where M is greater than or equal to 0, and M is an integer. The value of M is positively correlated with the screen refresh rate. In this embodiment, during the volume bar pop-up and volume bar collapse phases, and when the value of M is not set, the default value of M can be 0; or the electronic device can configure the value of M according to the screen refresh rate.
[0278] In this way, during the volume bar pop-up and volume bar collapse phases, electronic devices can reuse the blur layer of the previous frame image by skipping frames, reducing the number of times the GPU is called to generate the blur layer, thereby reducing GPU power consumption and load.
[0279] Figure 13 illustrates a flowchart of an embodiment of this application that adjusts the ratio of skipped frame images to copied frame images based on changes in the lower interface layer; as shown in Figure 13:
[0280] When the volume bar is stable, the electronic device can detect changes in the parameters of the lower interface layer. These parameters may include a third value and a fourth value. The third value can indicate a change in the display of the lower interface layer (e.g., the dynamic interface shown in Figure 4), while the fourth value can indicate no change in the display of the lower interface layer (e.g., the static interface shown in Figure 3). Specifically, the third value can be, for example, True, and the fourth value can be, for example, False; the third value can be, for example, 1, and the fourth value can be, for example, 0; or the third value can be, for example, 0, and the fourth value can be, for example, 1. This application does not limit the form or data of the changing parameters.
[0281] In the first implementation, when the change parameter of the lower interface layer is the third value, the electronic device can set the ratio of the skipped frame image to the copied frame image to be N, where N is greater than or equal to 1 and is an integer. The value of N can be a preset value, for example, the default value of N is 1.
[0282] In the second implementation, when the change parameter of the lower interface layer is the fourth value, the electronic device can use the skip frame image and not the copy frame image, until the change parameter of the lower interface layer is switched to the third value, at which point the electronic device uses the N value to set the skip frame image and the copy frame image.
[0283] Figure 13a shows a schematic diagram of the first implementation method, taking the video application window as the lower interface layer and the electronic device displaying the first to fifth frames of the image as an example.
[0284] During video playback, the video application window continuously changes, as shown in Figure 4c and Figure 4d. The corresponding image below the blur layer also changes accordingly; from the blur layer of the first frame to the blur layer of the fifth frame, the character gradually moves to the left.
[0285] The parameter for the lower interface layer is set to the first value. The electronic device uses the N value to set the skipped frame image and the copied image. Taking N=1 as an example, in Figure 13a, the first frame is a copied frame, the second frame is a skipped frame, the third frame is a copied frame, the fourth frame is a skipped frame, the fifth frame is a copied frame, and so on...
[0286] Figure 13b illustrates the principle of setting a skipped frame image when the lower interface layer changes.
[0287] During the volume bar stabilization phase, the refresh rate of the lower-level interface of the electronic device and the refresh rate of the volume bar are different; the refresh rate of the volume bar can be greater than the refresh rate of the lower-level interface; the refresh rate of the frame image is the same as the refresh rate of the volume bar.
[0288] Taking a lower-level interface refresh rate of 60Hz, a volume bar refresh rate of 120Hz, and a frame image refresh rate of 120Hz as an example, the electronic device refreshes the lower-level interface layer at a frequency of 60Hz; the lower-level interface layer corresponds to the lower-level interface layer of frame 1' to frame 4'. The electronic device refreshes the frame image at a frequency of 120Hz; the blur layer corresponds to the blur layer of frame 1 to frame 7.
[0289] Specifically, for each frame of the lower interface layer refreshed by the electronic device, two corresponding blur layers are refreshed. For example, when refreshing the 1'th frame of the lower interface layer, the 1st frame image is refreshed, and the blur layer is obtained by blurring and masking the 1'th frame of the lower interface layer. While displaying the 1'th frame of the lower interface layer, the electronic device refreshes the 2nd frame image; since the electronic device does not refresh the lower interface layer, the blur layer of the 2nd frame image is the same as the blur layer of the 1st frame image, and the electronic device can reuse the blur layer; the 2nd frame image can be a skipped frame image.
[0290] When refreshing the lower interface layer of frame 2', the image of frame 3 is refreshed. The blurred layer is obtained by blurring and masking the lower interface layer of frame 2'. During the display of the lower interface layer of frame 2', the electronic device refreshes the image of frame 4. Since the electronic device does not refresh the lower interface layer, the blurred layer of frame 4 is the same as the blurred layer of frame 3, and the electronic device can reuse the blurred layer; frame 4 can be a skipped frame image... and so on.
[0291] It should be noted that, in the embodiments of this application, the change parameters of the lower interface layer obtained by the electronic device can be used to describe whether there is a change between the lower interface in the previous frame and the lower interface in the next frame. For example, if the image of the lower interface layer in frame 1' is different from the image of the lower interface layer in frame 2', the electronic device can detect that the change parameter of the lower interface layer is a third value.
[0292] In this way, when the volume bar is stable and the underlying interface changes, the electronic device can reduce the number of times the GPU is called to generate a blur layer by adjusting the ratio of the skip frame image to the copy frame image, thereby reducing GPU power consumption and load. In addition, even if the underlying interface changes, reusing the blur layer of the copy frame in the skip frame image will not affect the smoothness of the user's viewing of the frame image.
[0293] Figure 13c illustrates the second implementation method, taking the lower interface layer as a social application window and the electronic device displaying the first to fifth frames of images as an example.
[0294] During social chat, the social application window can remain unchanged for a period of time. For example, in the interface shown in Figure 3c, if the electronic device does not receive a new chat message, the interface shown in Figure 3d can be the same as the interface shown in Figure 3c. The images of the lower layers corresponding to the blurred layers are also the same; for example, the blurred layers of the first frame image to the fourth frame image all include the user's avatar, and the content and position of the user's avatar are the same.
[0295] After an electronic device uses a copied frame, if no change is detected in the underlying interface layer (e.g., the change parameter of the underlying interface layer is the fourth value), the electronic device can continue to use the skipped frame image. For example, the first frame image is a copied frame image, the second frame image is a skipped frame image, the third frame image is a skipped frame image, and the fourth frame image is a skipped frame image.
[0296] When the underlying interface layer changes (e.g., the parameter for the underlying interface layer changes to the third value), the electronic device stops using the skipped frame image and updates the blurred layer with a copied frame image. For example, when displaying the 5th frame image, the electronic device receives a new chat message, the user's avatar position changes, and the 5th frame image is a copied frame image.
[0297] In this way, when the volume bar is stable and the lower interface has not changed, the electronic device can set up multi-frame skipping images. All multi-frame skipping images reuse the blur layer of the previous frame copy image, further reducing the number of times the GPU is called to generate blur layers, thus reducing GPU power consumption and load.
[0298] In this embodiment, during the display of a frame image, the frame image can have multiple states, including a closed state, a normal frame state, a copied frame state, and a skipped frame state. The closed state can be understood as the state where the image synthesis frame image is processed using the native process (e.g., the process shown in Figure 2). For example, in the closed state, the electronic device does not use copied or skipped frame images. The normal frame state can be the state corresponding to the first frame image after the image synthesis method provided in this embodiment is started; in the normal frame state, the electronic device can create a blur buffer. In the copied frame state, the electronic device can call the screenshot blur interface to generate a blur layer in the blur buffer; in the skipped frame state, the electronic device can reuse the blur layer in the blur buffer.
[0299] Figure 14 shows a flowchart of an image synthesis method provided in an embodiment of this application, as shown in Figure 14:
[0300] S1401, The electronic device acquires multiple layers of the current frame image; the current frame image is in the off state.
[0301] In the off state, the electronic device does not use HWC to process frame images of real-time blurred scenes. For example, the current frame image can be the frame image to be displayed when the first operation is received; the previous frame image can be the frame image before the first operation was received.
[0302] Taking volume adjustment as an example, the process of the volume bar appearing and disappearing can be shown in Figure 15: the previous frame image can be the 0th frame image, and the previous frame image can be, for example, the interface shown in Figure 3a. The 0th frame image is in the stage where the volume bar is not displayed, and the state of the 0th frame image is off. In the off state, the electronic device adopts the process shown in Figure 2a, and uses HWC to synthesize the 0th frame image; the scheme of synthesizing copy frame images or skipping frame images shown in the embodiments of this application is not enabled, and the electronic device does not execute the steps shown in Figures 8 and 10.
[0303] In Figure 15, the current frame image can be the first frame image received when the first operation is received. The initial state of the current frame image is the same as the state of the previous frame image; for example, the initial state of the first frame image is the off state. The electronic device can acquire multiple layers of the first frame image and determine the state of the first frame image.
[0304] S1402, The electronic device determines whether the preset conditions are met.
[0305] The preset conditions are used to determine whether the image compositing method provided in this application embodiment is applicable. If the preset conditions are met, the electronic device uses the image compositing method provided in this application embodiment to compose the frame image; if the preset conditions are not met, the electronic device uses the native process to compose the frame image.
[0306] In this embodiment of the application, after meeting preset conditions, the electronic device can use copied frame images and skipped frame images in a real-time blurred scene. The preset conditions may be, for example, that there is one and only one target layer among the multiple layers of the current frame image, the target layer is a preset layer, and the lower interface layer does not include a security layer.
[0307] In this embodiment, the electronic device may include a preset layer list; the electronic device can search by layer name to determine whether the target layer is a preset layer. Preset layers may include, for example, a volume bar layer, a message pop-up layer, and a sidebar layer.
[0308] It is understandable that HWC is suitable for relatively simple image compositing processes. When there are many target layers, the calculation process becomes more complex, and the computing power of HWC may not support the simultaneous processing of multiple real-time blurred scenes. Therefore, this application embodiment sets the preset condition to have one and only one target layer. When the target layer is not included among multiple layers or the number of target layers is greater than one, the electronic device does not use the image compositing method provided in this application embodiment.
[0309] It should be noted that the number of target layers can also be set according to the actual computing power of HWC in this embodiment. This embodiment does not impose any limitations on this.
[0310] The image compositing method provided in this application relies on a screenshot blurring interface to generate a blurred layer. The security layer has a high security level, and electronic devices typically do not allow screenshots of the security layer. When a security layer is included in the lower interface layer, the electronic device may not need to use the image compositing method provided in this application.
[0311] In this embodiment, if preset conditions are met, the electronic device can execute step S1403. If the preset conditions are not met, the electronic device can execute step S1401, without using the image synthesis method provided in this embodiment.
[0312] S1403. Under the condition that the preset conditions are met, the electronic device sets the state of the current frame image to the normal frame state.
[0313] In the case where there is one and only one target layer among multiple layers of the current frame image, and this target layer is a preset layer, and the lower interface layer is not a safe layer, the electronic device uses the image compositing method provided in this application embodiment. The electronic device updates the state of the current frame image to a normal frame state. In the normal frame state, the electronic device can use HWC to compose the frame image. The number of frame images in the normal frame state is 1; the electronic device only sets the first frame image after enabling the image compositing method provided in this application embodiment as a frame image in the normal frame state.
[0314] S1404, Electronic device creates a blur buffer and uses HWC to composite multiple layers.
[0315] It should be noted that the electronic device needs a certain amount of time to create the blur buffer. If the process shown in Figure 8 is executed immediately after the preset conditions are met, it is necessary to wait for the blur buffer to be created before compositing the current frame image; the rendering delay of the current frame image is relatively long, which may affect the smoothness of the user interface. Therefore, in this embodiment, after the preset conditions are met, the electronic device can synchronously create the blur buffer during the processing of the first frame image, so that the blur buffer can be used by subsequent copied frame images and skipped frame images.
[0316] Furthermore, after detecting a target layer, the electronic device can set a normal frame state; the target layer carries blur parameters. In a typical scenario, after detecting blur parameters, the electronic device instructs the GPU to perform compositing; however, GPU compositing causes the electronic device to switch compositing methods, resulting in a flickering phenomenon where the current frame image suddenly darkens and the next frame image suddenly brightens. Therefore, in this embodiment, when the current frame image is in a normal frame state, the electronic device can mask the blur parameters in the target layer; when the target layer of the current frame image does not contain blur parameters, the electronic device can instruct the use of HWC compositing.
[0317] When the target layer does not include blur parameters, electronic devices can use HWC to composite multiple layers. For example, the HWC compositing process might involve the electronic device first drawing the lower-level interface layer onto a buffer based on the hierarchical relationship, and then drawing the status bar layer and volume bar layer onto the buffer containing the lower-level interface layer.
[0318] It should be noted that during the HWC process of compositing normal frame images, the electronic device begins to create a blur buffer. Therefore, the normal frame image does not include the blur layer cached through the blur buffer. Normal frame images appear, for example, in the first frame image stage where the volume bar appears from nothing, or the electronic device displays a small volume bar or no volume bar at all. In the first frame image, the electronic device does not draw a blur layer; however, because the volume bar is small or absent in the user interface, the user is unaware that the first frame image lacks a blur layer.
[0319] In this way, the electronic device can be controlled to transition from a non-real-time blurred scene to a real-time blurred scene through normal frame images. A blur buffer layer is created in the normal frame state for subsequent HWC synthesis of the real-time blurred scene image. In addition, by masking the blur parameters, the normal frame is also controlled to use HWC synthesis, so that the electronic device uses HWC synthesis throughout the entire process from non-real-time blurred scene to real-time blurred scene, thereby avoiding flickering problems caused by inconsistent brightness.
[0320] After synthesizing and displaying the current frame image, the electronic device can acquire multiple layers of the next frame image and update the next frame image to the current frame image. For example, the second frame image in Figure 15 can be updated to the current frame image.
[0321] S1405. The electronic device determines whether the preset conditions are met.
[0322] This step can be referred to in the relevant description in step S1402, and will not be repeated here.
[0323] In this embodiment, if preset conditions are met, the electronic device can execute step S1406. If preset conditions are not met, the electronic device can execute step S1401, setting the state of the current frame image to a closed state to stop using the image synthesis method provided in this embodiment.
[0324] S1406. Under the condition that the preset conditions are met, the electronic device sets the state of the current frame image to the copy frame state.
[0325] Under preset conditions, the electronic device can set the state of the current frame image to a copy frame state. In the copy frame state, the electronic device can execute, for example, the process shown in Figure 8, such as step S1407.
[0326] S1407. The electronic device uses the GPU to generate a blurred layer in the blurred buffer and uses HWC to composite the blurred layer and multiple layers.
[0327] This process corresponds to the relevant descriptions in the embodiments shown in Figures 8 and 9, and will not be repeated here.
[0328] After synthesizing and displaying the current frame image, the electronic device can acquire multiple layers of the next frame image and update the next frame image to the current frame image.
[0329] S1408. The electronic device determines whether the preset conditions are met.
[0330] The process can be referred to in the relevant description in step S1405, and will not be repeated here.
[0331] S1409. Under the condition that the preset conditions are met, the electronic device determines whether the target layer has changed.
[0332] The transition phase can be a pop-up phase or a collapse phase. A pop-up phase can be, for example, a volume bar pop-up phase (e.g., frames 2 to 5 in Figure 15), a notification message pop-up phase, or a sidebar pop-up phase. A collapse phase can be, for example, a volume bar collapse phase (e.g., frames 11 to 14 in Figure 15), a notification message collapse phase, or a sidebar collapse phase. During the transition phase, the size of the target layer changes, and the size of the blurred layer also changes accordingly. The change parameter for the blurred layer can be a first value.
[0333] The target layer is not in a changing phase, for example, it may be in a stable phase, such as the volume bar stable phase (e.g., frames 6 to 10 in Figure 15), the notification message stable phase, or the sidebar stable phase. When in a stable phase, the size of the target layer no longer changes, and the size of the blurred layer also no longer changes; the change parameter for the blurred layer can be a second value.
[0334] In this embodiment of the application, when the target layer is in a changing phase, the electronic device may execute step S1410; when the target layer is in a stable phase, the electronic device may execute step S1414.
[0335] S1410. When the target layer is in a change phase, the electronic device determines whether the screen refresh rate is greater than the first refresh rate threshold.
[0336] When the change parameter of the blur layer can be a first value, the electronic device can adjust the ratio of the skipped frame image and the copied frame image according to the screen refresh rate, as described in the embodiment shown in Figure 12. Specifically, in a high screen refresh rate scenario, the electronic device can use both the skipped frame image and the copied frame image, as shown in step S1411. In a low screen refresh rate scenario, the electronic device can use the copied frame image and not the skipped frame image, as shown in step S1413.
[0337] S1411. When the screen refresh rate is greater than the first refresh rate threshold, the electronic device sets the state of the current frame image to frame skipping state.
[0338] The current frame image can be, for example, the third frame image in Figure 15. At a high screen refresh rate, the third frame image can be a skipped frame image. In the skipped frame state, the electronic device can execute, for example, the process shown in Figure 10, such as step S1412.
[0339] S1412. Electronic devices use HWC to composite blurred layers and multiple layers.
[0340] The blurred layer is the blurred layer of the previous copied frame image. This process can be referred to the relevant description in the embodiment shown in Figure 10, and will not be repeated here.
[0341] Optionally, the electronic device can set a threshold for the number of frame skips. When the target layer is in a changing phase, the electronic device can set the threshold to an M value.
[0342] After step S1412, the electronic device can acquire multiple layers of the next frame image and update the next frame image to the current frame image. When the previous frame image is in a frame skipping state, the electronic device can determine whether the count threshold has been reached. If the count threshold is reached, the electronic device sets the current frame image to a copy frame state and executes step S1407. If the count threshold is not reached, the electronic device sets the current frame image to a frame skipping state and executes step S1412.
[0343] For example, in Figure 15, the electronic device acquires multiple layers of the fourth frame image and updates the fourth frame image to the current frame image. The third frame image (the previous frame image) is in a frame skipping state; when the number of times reaches the threshold, the fourth frame image is set to a copy frame state; when the number of times does not reach the threshold, the fourth frame image is set to a frame skipping state.
[0344] In this way, in scenarios where the target layer is in a changing phase and the screen refresh rate is high, electronic devices can alternately display skipped frame images and copied frame images according to the ratio of skipped frame images to copied frame images.
[0345] Alternatively, S1413, if the screen refresh rate is less than or equal to the first refresh rate threshold, the electronic device sets the state of the current frame image to the copy frame state.
[0346] The current frame image can be, for example, the third frame image in Figure 15. At a low screen refresh rate, the third frame image can be a copy frame image. After being set to copy frame state, the electronic device can perform step S1407.
[0347] For example, in Figure 15, the electronic device acquires multiple layers of the fourth frame image and updates the fourth frame image to the current frame image. The third frame image (the previous frame image) is in the copy frame state, and the electronic device sequentially goes through S1408, S1409, S1410, S1413 and S1407 to process the fourth frame image into a copy frame image.
[0348] In this way, in scenarios where the target layer is in a changing phase and the screen refresh rate is low, electronic devices can use copied frame images instead of skipped frame images.
[0349] Steps S1410-S1413 show the frame image state during the volume bar pop-up or volume bar collapse phases; the frame image state during the volume bar stabilization phase is described below.
[0350] Following step S1409, the method further includes:
[0351] S1414. When the target layer is in a stable phase, the electronic device sets the state of the current frame image to frame skipping state.
[0352] When the change parameter of the blurred layer can be a second value, the electronic device can adjust the ratio of the skipped frame image and the copied frame image according to the change parameter of the lower interface layer, as described in the embodiment shown in Figure 13.
[0353] For example, in Figure 15, the previous frame image is the 5th frame image, which is a copied frame image; the current frame image is the 6th frame image, which is a skipped frame image, thus entering a loop of alternating display between copied frames and skipped frames.
[0354] S1412. Electronic devices use HWC to composite blurred layers and multiple layers.
[0355] After synthesizing and displaying the current frame image, the electronic device can acquire multiple layers of the next frame image and update the next frame image to the current frame image. For example, the 7th frame image in Figure 15 can be updated to the current frame image.
[0356] S1415. The electronic device determines whether the preset conditions are met.
[0357] The process can be referred to in the relevant description in step S1405, and will not be repeated here.
[0358] S1416. Under the condition that the preset conditions are met, the electronic device determines whether the lower interface layer has changed.
[0359] The electronic device acquires the change parameters of the lower interface layer and determines whether the lower interface layer has changed based on these parameters. Specifically, when the change parameter of the lower interface layer is the third value, the image of the lower interface layer changes, as in step S1417. When the change parameter of the lower interface layer is the fourth value, the image of the lower interface layer does not change, as in step S1418.
[0360] S1417. When the lower interface layer changes, the electronic device sets the state of the current frame image to the copy frame state.
[0361] The change parameter of the lower interface layer is the third value. The electronic device determines that the lower interface layer has changed and sets the state of the current frame image to copy frame state.
[0362] In some embodiments, a change in the lower interface layer can be: the lower interface layer corresponding to the previous few frames remains unchanged, while the lower interface layer corresponding to the current frame changes. Refer to Figure c in Figure 13 for the scene shown in frames 2 through 5.
[0363] For example, the first few frames are all skipped frames. After detecting a change in the lower interface layer, the current frame image is processed into a copy frame image to update the blur layer.
[0364] In other embodiments, the change in the lower interface layer can be: the lower interface layer corresponding to the previous few frames changes, and the lower interface layer corresponding to the current frame also changes. Refer to Figure a in Figure 13, showing the scene from the second to the third frame.
[0365] Optionally, the electronic device can set a threshold for the number of frame skips. When the changing parameters of the underlying interface layer corresponding to multiple consecutive frames are all of the third value, the electronic device can set the threshold for the number of frame skips to a value of N.
[0366] For example, after detecting a change in the underlying interface layer, the electronic device determines whether the frame skipping threshold has been reached. If the previous frames were all skipped frames and the threshold N has been reached, the electronic device processes the current frame as a copy frame to update the blurred layer.
[0367] Alternatively, S1418, if the lower interface layer has not changed, the electronic device sets the state of the current frame image to frame skipping state.
[0368] The change parameter of the lower interface layer is the fourth value. The electronic device determines that the lower interface layer has not changed and sets the state of the current frame image to frame skipping state.
[0369] In some embodiments, "the lower interface layer has not changed" can mean that the lower interface layer corresponding to the previous few frames has not changed, and the lower interface layer corresponding to the current frame has also not changed. Refer to the scene shown in the second to fourth frames in Figure c of FIG13.
[0370] For example, the previous frame image can be a skipped frame image or a copied frame image. When the underlying interface layer is not detected to have changed, the electronic device processes the current frame image as a skipped frame image to reuse the blurred layer.
[0371] In this way, the electronic device synthesizes frame images according to their state. After displaying a normal frame image, the electronic device can use copied and skipped frame images to avoid forcibly synthesizing images through the GPU, reducing the number of times the GPU generates blur layers, thereby reducing GPU power consumption, load, and rendering latency. In addition, the electronic device synthesizes copied and skipped frame images through HWC, and without involving HWC and GPU switching, the electronic device will not exhibit a bright-dark-bright flickering phenomenon, improving the user experience.
[0372] It should be noted that this application embodiment only uses a real-time blurred scene as an example to illustrate the process of using HWC to composite images. Real-world scenarios may also include other scenarios that force GPU image compositing, such as real-time shadow scenes and real-time rounded corner scenes. In these scenarios, the electronic device can also break down the processing steps. For example, the GPU performs shadow processing or rounded corner processing on layers and stores the processing results in a pre-created buffer; HWC then calls the processing results in the buffer and composites them with other layers. This can reduce the amount of GPU overlay processing used in image processing, thereby reducing GPU load, power consumption, and rendering latency. This application embodiment does not limit the scenarios where HWC compositing can replace GPU compositing.
[0373] Based on the above embodiments, this application provides an image compositing method, as shown in FIG16. This method can be applied to an electronic device, which may include an HWC. The image compositing method may include the following steps:
[0374] For example, the interface described in the embodiments of this application can be understood as a user interface (UI). The UI refers to the layout design of the software's human-computer interaction, operation logic, and aesthetics. Taking the Android system as an example, the interface described in the embodiments of this application can be understood as an Activity interface, which can be an interface without navigation.
[0375] It is understandable that changes in the layout design of the interface will result in a new interface. For example, the first interface is one that does not display the blur layer and the target layer; for instance, the interfaces shown in Figure 3a, Figure 3f, Figure 4a, and Figure 4f do not include the target layer carrying blur parameters, such as volume bars, message pop-ups, or sidebars. The second interface is one that displays the blur layer and the target layer; for instance, the interfaces shown in Figure 3b to Figure 3e, and Figure 4b to Figure 4e. In this embodiment, the first interface can be used to describe one or more frame images that do not include the blur layer and the target layer, and the content of these frame images can be the same or different; the second interface can be used to describe one or more frame images that include the blur layer and the target layer, and the content of these frame images can be the same or different.
[0376] Specifically, S1601, display the first interface, which includes a first layer but does not include a blurred layer, and the size of the first layer is the same as the screen size of the electronic device; the first interface is composited using HWC.
[0377] The first interface can be the interface displayed before the target layer (such as a volume bar, message pop-up, or sidebar) pops up, as shown in Figure 3a or Figure 4a. The first layer can be a lower interface layer, such as the social application window layer in Figure 3 or the video application window layer in Figure 4; the first layer can be a full-screen layer, with the same size as the screen of the electronic device.
[0378] During the process of not displaying the target layer, multiple layers do not involve real-time blurring. The electronic device uses the native process to composite the image of the first interface using HWC, as shown in Figure 2.
[0379] S1602, Receive the first operation in the first interface.
[0380] For the first operation, please refer to the relevant description in step S801.
[0381] S1603. According to the first operation, a second interface is displayed. The second interface includes a first layer, a second layer, and a blur layer corresponding to the second layer. The blur layer is obtained from the blur buffer. If the second layer is a preset layer and carries blur parameters, the second interface is composed of the first layer, the second layer, and the blur layer by HWC. The preset layer includes any of the following: volume bar layer, message pop-up layer, or sidebar layer. The blur layer is located above the first layer, and the second layer is located above the blur layer.
[0382] The second interface can be the interface displayed during the pop-up, stable, and collapsed phases of the target layer (such as volume bar, message pop-up, or sidebar), for example, the interfaces shown in Figure 3b to Figure 3e, and the interfaces shown in Figure 4b to Figure 4e.
[0383] The first layer can be the lower interface layer; the second layer can be the target layer, such as a volume bar layer, a message pop-up layer, or a sidebar layer. The blur layer corresponds to the second layer; for example, the size of the blur layer is the same as the size of the volume bar layer, and the position of the blur layer is the same as the position of the volume bar layer.
[0384] The second interface may include multiple frames of images. Any frame of the multiple frames in the first interface may be a copy frame image provided in the embodiments of this application, or it may be a skipped frame image. The blur layer is obtained from the blur buffer. This can be understood as follows: during the process of composing the second interface, when the second interface is a copy frame image, the electronic device can generate a blur layer of the copy frame image and store it in the blur buffer; and use the blur layer in the blur buffer to compose the copy frame image. When the second interface is a skipped frame, the electronic device can reuse the blur layer of the previous copy frame image in the blur buffer.
[0385] After obtaining the blurred layer, the electronic device does not need to call the GPU for blurring, so the electronic device can use HWC to synthesize multiple frames of images from the second interface.
[0386] It should be noted that the image compositing method provided in this application is applicable to real-time blurred scenes. Electronic devices can determine whether to use HWC to process real-time blurred scenes by using preset layers and blur parameters. For example, when the second layer is a preset layer and carries blur parameters, the HWC compositing method provided in this application is used, as shown in the embodiments of Figure 8 or Figure 10.
[0387] S1604. After the second layer disappears, the first interface is displayed.
[0388] It is understood that in this embodiment of the application, only the first interface is described, excluding the blurred layer and the target layer; no restrictions are placed on the changes of the first layer in the first interface. For example, the first interface in step S1601 and the first interface in S1604 are both interfaces that do not include the blurred layer and the target layer; but step S1601
[0389] The content of the first layer in the first interface can be the same as or different from the content of the first layer in the first interface of S1604.
[0390] In this step, the first interface can be the interface displayed after the target layer (such as volume bar, message pop-up or sidebar) is collapsed, as shown in f in Figure 3 or f in Figure 4.
[0391] During the process of not displaying the target layer, multiple layers do not involve real-time blurring. The electronic device uses the native process to composite the image of the first interface using HWC, as shown in Figure 2.
[0392] In this way, the electronic device uses HWC compositing when the target layer is not displayed; the electronic device can still use HWC compositing when the target layer is displayed, reducing the switching process between HWC compositing and GPU compositing; it can reduce the power consumption, load and rendering latency of the electronic device when forced to use GPU compositing in real-time blurred scenes; at the same time, it reduces the flickering problem of light-dark-light caused by switching compositing methods.
[0393] In one possible implementation, the electronic device further includes a GPU; the second interface includes a first image; displaying the second interface includes: acquiring multiple layers of the first image, the multiple layers of the first image including a first layer and a second layer of the first image; the GPU blurring the first layer of the first image according to blur parameters to obtain a blurred layer of the first image; the blurred layer of the first image is stored in a blur buffer; the HWC composites the blurred layer of the first image, the first layer of the first image, and the second layer of the first image to obtain the first image; and displaying the first image.
[0394] The second interface may include multiple frames of images, such as a first image. The first image can be understood as the copy frame image shown in the embodiments of this application. Refer to the relevant description of the synthesis process of the first frame image in the embodiment shown in FIG8.
[0395] Understandably, the lifecycle of the blur buffer is the same as that of the target layer (corresponding to the second layer). This allows the GPU-generated blur layer to be stably stored in the blur buffer, facilitating the use of HWC compositing during image compositing and the reuse of the blur layer when compositing adjacent frames. This reduces the power consumption, load, and rendering latency of electronic devices when forcing GPU compositing in real-time blurred scenes; simultaneously, it reduces the flickering issue caused by switching between bright and dark compositing modes.
[0396] In one possible implementation, the second interface further includes a second image, which is the next frame image of the first image; after displaying the first image, it further includes: acquiring multiple layers of the second image, the multiple layers of the second image including a first layer of the second image and a second layer of the second image.
[0397] It is understandable that the first image is a copied frame image, and the second image can be either a skipped frame image or a copied frame image. When the target layer is in the pop-up or collapse phase, the electronic device can determine whether the second image is a skipped frame or a copied frame based on the screen refresh rate.
[0398] In some embodiments, if the size or position of the second layer of the second image and the second layer of the first image are different, and the screen refresh rate is greater than the refresh rate threshold, the blur layer of the first image in the blur buffer is not updated; the blur buffer stores the blur layer of the first image.
[0399] The difference in size or position between the second layer of the second image and the second layer of the first image can be used to characterize that the target layer is in a changing phase, such as a pop-up phase or a collapse phase.
[0400] The refresh rate threshold can be a threshold used to determine whether to skip frames. The refresh rate threshold can be the lowest refresh rate threshold among the multiple refresh rate thresholds shown in the embodiments of this application, such as the first refresh rate threshold.
[0401] When the screen refresh rate is greater than the refresh rate threshold, the electronic device synthesizes a second image by reusing the blur layer. The second image is a frame skipping image, such as the second frame image, the fourth frame image, and the sixth frame image shown in Figure 12a.
[0402] In Figure 14, the first image may correspond to the copy frame image synthesized in steps S1406 and S1407. The second image may correspond to the skip frame image synthesized in steps S1408, S1409, S1410, S1411 and S1412.
[0403] In other embodiments, if the screen refresh rate is less than or equal to a refresh rate threshold, then the second image is a copy frame image. The electronic device can use the GPU to blur the first layer of the second image according to blur parameters to obtain a blurred layer of the second image; the blurred layer of the second image is then updated in the blur buffer. The second image can be, for example, the 2' frame image, the 3' frame image, the 4' frame image, etc., as shown in Figure 12b.
[0404] In Figure 14, the first image may correspond to the copy frame image synthesized in steps S1406 and S1407. The second image may correspond to the copy frame image that is cycled back to steps S1406 and S1407 via steps S1408, S1409, S1410, and S1413.
[0405] In this way, during the target layer pop-up and collapse phases, electronic devices can reuse the blurred layer of the previous frame image by skipping frames, reducing the number of times the GPU is called to generate the blurred layer, thus reducing GPU power consumption and load; and by using HWC compositing, brightness changes caused by switching compositing methods can be avoided.
[0406] In one possible implementation, while the target layer is in a change phase and after the second image is displayed, the second interface also includes a third image, where the first image is the a-th frame image, the second image is the (a+1)-th frame image, and the third image is the b-th frame image, where a is less than b, and both a and b are positive integers.
[0407] When the target layer changes and the screen refresh rate is high, electronic devices can alternate between copy frame images and skip frame images. For example, after processing one copy frame image, one or more skip frame images can be processed. One or more skip frame images can be, for example, a second image to a (b-1)th frame image; the electronic device can composite the images using a blurred layer of the first image (e.g., the previous copy frame image) via HWC.
[0408] During the synthesis of the second image up to the (b-1)th frame, the blur layer of the first image in the blur buffer is not updated. Then, the third image is used as the current frame image, and the processing may include:
[0409] The process involves acquiring multiple layers of a third image, including a first layer and a second layer. If the second layer of the third image differs in size or position from the second layer of the first image (corresponding to the target layer being in a transition phase), and the screen refresh rate is greater than a refresh rate threshold, it is determined whether the number of times the blurred layer of the first image has been used has reached a first threshold (a preset threshold for the number of frame skips when the target layer is in a transition phase, such as the M value). If the first threshold is reached, the GPU blurs the first layer of the third image according to the blur parameters to obtain a blurred layer of the third image; the blurred layer of the third image is updated in the blur buffer; HWC composites the blurred layer, the first layer, and the second layer of the third image to obtain the third image; and the third image is then displayed.
[0410] After one or more frame skips, the electronic device can use copied frames to update the blur layer in the blur buffer. This scenario corresponds to the 3rd, 5th, and 7th frames in Figure 12a.
[0411] In this way, during the target layer pop-up and target layer collapse phases, the electronic device can reuse the blurred layer of the previous frame image through the M-frame skipping image, reducing the number of times the GPU is called to generate the blurred layer, thus reducing GPU power consumption and load. At the same time, after a period of time (e.g., when the number of times reaches a threshold), the blurred layer is updated to improve the similarity and accuracy between the blurred layer of the previous frame copy image and the blurred layer of the current frame image.
[0412] In one possible implementation, after the target layer is in a stable phase and after the first image (copy frame image) is displayed, the second interface also includes a fourth image, where the first image is the c-th frame image and the fourth image is the d-th frame image, where c is less than d and both c and d are positive integers, meaning that the fourth image is displayed after the first image.
[0413] When the target layer is in a stable phase, the electronic device can set frame skipping and copying frames according to the changes in the lower interface layer. For example, the second layer in frames c+1 to d-1 is the same size and position as the second layer of the first image (corresponding to the target layer being in a stable phase); the first layer in frames c+1 to d-1 is the same as the first layer of the first image (corresponding to the lower interface not changing). In this case, frames c+1 to d-1 can be skipped frames such as the second, third, and fourth frames in Figure c of Figure 13; the skipped frames are all synthesized by HWC using the blurred layer of the first image (the previous copy frame image).
[0414] The process of updating the blurred layer in the blur buffer can be, for example, after displaying the first image, further including: obtaining multiple layers of the fourth image, the multiple layers of the fourth image including the first layer of the fourth image and the second layer of the fourth image; the second layer of the fourth image and the second layer of the first image have the same size or position (corresponding to the target layer being in a stable stage), and the first layer of the fourth image and the first layer of the first image have different images (corresponding to the lower interface layer changing images); in this case, the GPU performs blur processing on the first layer of the fourth image according to the blur parameters to obtain the blurred layer of the fourth image; and updates the blurred layer of the fourth image in the blur buffer.
[0415] The fourth image can be, for example, the fifth frame image in Figure 13c. Frame skipping is used when the lower interface layer remains unchanged for the first few frames; frame copying is used after the lower interface layer changes.
[0416] In this way, when the volume bar is stable and the lower interface has not changed, the electronic device can set up multi-frame skipping images. All multi-frame skipping images reuse the blur layer of the previous frame copy image, further reducing the number of times the GPU is called to generate blur layers, thus reducing GPU power consumption and load.
[0417] In one possible implementation, after the target layer is in a stable phase and after displaying the first image (copy frame image), the second interface also includes a fifth image, which is the next frame image of the first image. After displaying the first image, the interface further includes: acquiring multiple layers of the fifth image, including a first layer and a second layer of the fifth image; when the second layer of the fifth image and the second layer of the first image have the same size or position (corresponding to the target layer being in a stable phase), and the first layer of the fifth image and the first layer of the first image have different images (corresponding to a change in the image of the lower interface layer), HWC composites the blurred layer of the first image, the first layer of the fifth image, and the second layer of the fifth image to obtain the fifth image; the blurred layer of the first image is stored in the blur buffer; and the fifth image is displayed.
[0418] It is understandable that the first layer of images in two adjacent frames are different, and the lower interface layer can be, for example, a video application window layer, as shown in Figure 13a or Figure 13b. When the target layer remains unchanged, but the lower interface layer changes, the electronic device can alternate between copy frames and skipped frames. For example, if the first image is a copy frame image, the fifth image in the next frame must be a skipped frame. Referring to Figure 14, the first image corresponds to steps S1406 and S1407; the fifth image corresponds to S1408, S1409, and S1414.
[0419] In this way, when the volume bar is stable and the underlying interface changes, the electronic device can reduce the number of times the GPU is called to generate a blur layer by adjusting the ratio of the skip frame image to the copy frame image, thereby reducing GPU power consumption and load. In addition, even if the underlying interface changes, reusing the blur layer of the copy frame in the skip frame image will not affect the smoothness of the user's viewing of the frame image.
[0420] In one possible implementation, the second interface also includes a sixth image, where the first image is the e-th frame image, the sixth image is the f-th frame image, e is less than f, and both e and f are positive integers; the sixth image can be the fifth image or a frame image after the fifth image. When the target layer remains unchanged and the lower interface layer changes, the electronic device can set one or more frame skips, and the one or more frame skip images can be, for example, the fifth image to the (f-1)-th frame images; the frame skip images are synthesized by HWC using the blurred layer of the first image (a copy of the previous frame image).
[0421] After displaying the multi-frame skipped image, the process also includes: acquiring multiple layers of the sixth image, including the first layer and the second layer of the sixth image; when the second layer of the sixth image and the second layer of the first image have the same size or position (corresponding to the target layer being in a stable phase), and the first layer of the sixth image and the first layer of the first image have different images (corresponding to the lower interface layer's image changing), determining whether the number of times the blurred layer of the first image has been used has reached a second threshold (corresponding to a preset threshold for the number of skipped frames when the target layer is in a stable phase, such as the N value); if the second threshold is reached, the GPU blurs the first layer of the sixth image according to the blur parameters to obtain the blurred layer of the sixth image; updating the blurred layer of the sixth image in the blur buffer; HWC composites the blurred layer, the first layer, and the second layer of the sixth image to obtain the sixth image; and displaying the sixth image.
[0422] In this way, during the target layer stabilization phase, the electronic device can reuse the blurred layer of the previous frame image through N-hop frame images, reducing the number of times the GPU is called to generate the blurred layer, thus reducing GPU power consumption and load. At the same time, after a period of time (e.g., when the number of times reaches a threshold), the blurred layer is updated to improve the similarity and accuracy between the blurred layer of the previous frame image and the blurred layer of the current frame image.
[0423] In one possible implementation, before displaying the second interface, the method further includes: in response to the first operation, acquiring multiple layers of the seventh image; the image of the first interface is the g-th frame image, and the seventh image is the g+1-th frame image, where g is a positive integer; the multiple layers of the seventh image include a first layer and a second layer of the seventh image; under the condition of satisfying a first preset condition, creating a blur buffer; simultaneously, masking the blur parameters, and using HWC to composite the first layer and the second layer of the seventh image to obtain the seventh image; wherein, during the process of compositing the seventh image, the blur buffer does not include the blur layer; the first preset condition includes: the second layer carries blur parameters, the second layer is a preset layer, the number of second layers is 1, and the first layer is not a safe layer.
[0424] This process corresponds to steps S1402 and S1403 in Figure 14. It is understood that the electronic device also sets up a normal frame image before displaying the copied frame image. The seventh image can be the first frame image received after the first operation, i.e., the normal frame image. The first preset condition corresponds to the preset condition in step S1402.
[0425] In this way, the electronic device can be controlled to transition from a non-real-time blurred scene to a real-time blurred scene through normal frame images. A blur buffer layer is created in the normal frame state for subsequent HWC synthesis of the real-time blurred scene image. In addition, by masking the blur parameters, the normal frame is also controlled to use HWC synthesis, so that the electronic device uses HWC synthesis throughout the entire process from non-real-time blurred scene to real-time blurred scene, thereby avoiding flickering problems caused by inconsistent brightness.
[0426] In one possible implementation, after receiving the first operation in the first interface, the method further includes: displaying a third interface according to the first operation; the third interface includes a first layer, a second layer, and a blurred layer corresponding to the second layer; the blurred layer is obtained from a temporary buffer of the GPU; if multiple layers of the third interface satisfy a second preset condition, the third interface is composited by the GPU from the multiple layers of the third interface; the second preset condition includes one or more of the following: the second layer carries a blur parameter, and the number of second layers is greater than 1; or, the second layer carries a blur parameter, and the preset layer does not include the second layer; or, the second layer carries a blur parameter, and the first layer is a safe layer.
[0427] It is understood that the first operation can be used to trigger a blurred scene; after receiving the first operation, the electronic device can perform HWC compositing on the image of the real-time blurred scene using the process shown in the above embodiments, provided that the first preset conditions are met. However, if the first preset conditions are not met, the electronic device can use the native process to process the image; for example, the electronic device can use HWC compositing or GPU compositing.
[0428] The second preset condition can be used to determine whether to use native GPU compositing. The second preset condition includes one or more of the following: the second layer carries a blur parameter, and the number of second layers is greater than 1 (for example, if HWC's processing power is insufficient to support the processing of multiple target layers, native GPU compositing can be used); or, the second layer carries a blur parameter, and the preset layers do not include the second layer (for example, if a full-screen blur layer covers the lower interface layer, the electronic device will not show the brightness change of the lower interface layer, and there is no flickering problem, so the electronic device can use native GPU compositing); or, the second layer carries a blur parameter, and the first layer is a safe layer (for example, if blur processing involves taking a screenshot of the lower interface layer, and the safe layer restricts screenshot permissions, the electronic device can use native GPU compositing).
[0429] In this way, if the first preset condition is not met, the electronic device can use the native process to synthesize the image to avoid the problem of insufficient HWC computing power or to avoid reducing the security of the security layer.
[0430] The flow of the image synthesis method provided in the embodiments of this application will be described below with reference to Figure 17.
[0431] This image synthesis method can be applied to electronic devices, including HWC and GPUs. The method includes:
[0432] S1701. Obtain multiple layers of the first interface; the multiple layers of the first interface include the first layer but do not include the blurred layer, and the size of the first layer is the same as the screen size of the electronic device.
[0433] S1702 and HWC combine multiple layers of the first interface to obtain the first interface.
[0434] The first interface is the interface before the target layer pop-up stage. Electronic devices can use HWC to composite the first interface.
[0435] S1703, Display the first interface.
[0436] S1704, Receive the first operation in the first interface.
[0437] S1705. Obtain multiple layers of the second interface; the multiple layers of the second interface include the first layer and the second layer.
[0438] S1706. Determine whether multiple layers of the second interface meet the first preset conditions; the first preset conditions include that the second layer carries a blur parameter, the second layer is a preset layer, the number of second layers is 1, and the first layer is not a safe layer; the preset layer includes any one of the following: volume bar layer, message pop-up layer, or sidebar layer.
[0439] Upon receiving the first operation, the electronic device can determine whether to use HWC to composite the real-time blurred scene. For example, if the first preset condition is met, the electronic device can use the HWC compositing process provided in this application embodiment to process the real-time blurred scene, as shown in steps S1707 and S1708; if the first preset condition is not met, the electronic device can use the native process to composite the image, as shown in steps S1709 and S1710.
[0440] S1707. Under the condition that the first preset condition is met, obtain the blurred layer corresponding to the second layer from the blurred buffer.
[0441] Under the condition that the first preset condition is met, the image on the second interface can be either a copied frame image or a skipped frame image; the blur layer of both types of frame images is obtained from the blur buffer.
[0442] S1708 and HWC combine the first layer of the second interface, the second layer of the second interface, and the blurred layer of the second interface to obtain the second interface; the blurred layer is located above the first layer, and the second layer is located above the blurred layer.
[0443] S1709, or, if the first preset condition is not met and the second preset condition is met, the GPU performs blurring processing on the first layer of the second interface to obtain a blurred layer; the blurred layer is stored in a temporary buffer; the second preset condition includes one or more of the following: the second layer carries blur parameters and the number of second layers is greater than 1; or, the second layer carries blur parameters and the preset layer does not include the second layer; or, the second layer carries blur parameters and the first layer is a safe layer.
[0444] S1710 and GPU combine the first layer of the second interface, the second layer of the second interface, and the blurred layer of the second interface to obtain the second interface.
[0445] S1711, Display the second interface.
[0446] It should be noted that steps S1701-S1711 can be referred to the relevant description in the embodiment shown in Figure 16, and will not be repeated here.
[0447] In this way, the electronic device uses HWC compositing when the target layer is not displayed; the electronic device can still use HWC compositing when the target layer is displayed, reducing the switching process between HWC compositing and GPU compositing; it can reduce the power consumption, load and rendering latency of the electronic device when forced to use GPU compositing in real-time blurred scenes; at the same time, it reduces the flickering problem of light-dark-light caused by switching compositing methods.
[0448] In one possible implementation, the second interface includes a first image and a second image; the method includes:
[0449] After obtaining the first layer and the second layer of the first image, the GPU blurs the first layer of the first image according to the blur parameters to obtain the blurred layer of the first image; the blurred layer of the first image is stored in the blur buffer; the blurred layer of the first image is obtained from the blur buffer; HWC combines the first layer, the second layer and the blurred layer of the first image to obtain the first image.
[0450] After obtaining the first layer of the second image and the second layer of the second image, the blurred layer of the first image is obtained from the blur buffer; HWC composites the first layer of the second image, the second layer of the second image, and the blurred layer of the first image to obtain the second image.
[0451] The first image can be a copied frame image. The electronic device can generate a blur layer during the synthesis of the first image and store the blur layer in a blur buffer. The second image can be a skipped frame image. The electronic device can reuse the blur layer of the first image (a copied frame from the previous frame) during the synthesis of the second image.
[0452] In this way, the electronic device can obtain a stable blurred layer through the blur buffer, enabling the synthesis of real-time blurred scene images via HWC and the reuse of blurred layers. When the blurred layers of two adjacent frames are the same or similar, the electronic device can skip the GPU blur processing. The electronic device uses the blurred layer of the previous frame as the blurred layer of the current frame, reducing the number of GPU blur processing operations and lowering GPU power consumption and load by reusing blurred layers. In addition, the electronic device uses HWC to synthesize multiple layers in the second frame image with the blurred layer, which reduces the forced GPU synthesis process for real-time blurred scenes, thereby reducing the bright-dark-bright flickering problem caused by switching synthesis methods.
[0453] Optionally, in the embodiment shown in Figure 17, the copy frame image and the skip frame image can be set as shown in Figure 16. This application will not elaborate further on this aspect.
[0454] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.
[0455] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0456] The image synthesis method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above image synthesis method.
[0457] As shown in Figure 18, the image synthesis apparatus 1800 can be used in communication equipment, circuits, hardware components, or chips. The image synthesis apparatus includes a display unit 1801 and a processing unit 1802. The display unit 1801 is used to support the display steps performed by the image synthesis apparatus 1800; the processing unit 1802 is used to support the information processing steps performed by the image synthesis apparatus 1800.
[0458] In a possible implementation, the image synthesis apparatus 1800 may also include a communication unit 1803. Specifically, the communication unit supports the image synthesis apparatus 1800 in performing data transmission and data reception steps. The communication unit 1803 may be an input or output interface, pins, or circuitry, etc.
[0459] In a possible embodiment, the image compositing apparatus may further include a storage unit 1804. The processing unit 1802 and the storage unit 1804 are connected via a line. The storage unit 1804 may include one or more memories, which may be devices in one or more devices or circuits used for storing programs or data. The storage unit 1804 may exist independently and be connected to the processing unit 1802 of the image compositing apparatus via a communication line. Alternatively, the storage unit 1804 may be integrated with the processing unit 1802.
[0460] Storage unit 1804 may store computer-executable instructions for the methods in the terminal device, so that processing unit 1802 executes the methods in the above embodiments. Storage unit 1804 may be a register, cache, or RAM, etc., and storage unit 1804 may be integrated with processing unit 1802. Storage unit 1804 may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, and storage unit 1804 may be independent of processing unit 1802.
[0461] The image synthesis method provided in this application can be applied to electronic devices with communication functions. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-described related information, which will not be repeated here.
[0462] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.
[0463] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.
[0464] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.
[0465] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0466] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.
[0467] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.
[0468] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.
Claims
1. An image compositing method characterized by, Applied to an electronic device, the electronic device including a hardware hybrid renderer (HWC), the method includes: The first interface is displayed, which includes a first layer but does not include a blur layer. The size of the first layer is the same as the screen size of the electronic device. The first interface is composited using the HWC. Receive the first operation in the first interface; According to the first operation, a second interface is displayed. The second interface includes the first layer, the second layer, and the blurred layer corresponding to the second layer. The blurred layer is obtained from the blurred buffer. When the second layer is a preset layer and carries blurred parameters, the second interface is composited by the HWC using the first layer, the second layer, and the blurred layer. The preset layer includes any one of the following: a volume bar layer, a message pop-up layer, or a sidebar layer. The blurred layer is located above the first layer, and the second layer is located above the blurred layer. After the second layer disappears, the first interface is displayed.
2. The method of claim 1, wherein, The electronic device further includes a graphics processing unit (GPU); the second interface includes a first image; the display of the second interface includes: Obtain multiple layers of the first image, wherein the multiple layers of the first image include the first layer of the first image and the second layer of the first image; The GPU performs blurring processing on the first layer of the first image according to the blur parameters to obtain the blurred layer of the first image; the blurred layer of the first image is stored in the blur buffer; The HWC composites the blurred layer of the first image, the first layer of the first image, and the second layer of the first image to obtain the first image; Display the first image.
3. The method of claim 2, wherein, The second interface also includes a second image, which is the next frame of the first image; after displaying the first image, it further includes: Obtain multiple layers of the second image, wherein the multiple layers of the second image include the first layer of the second image and the second layer of the second image; If the size or position of the second layer of the second image is different from that of the second layer of the first image, and the screen refresh rate is greater than the refresh rate threshold, the blurred layer of the first image in the blurred buffer is not updated; the blurred buffer stores the blurred layer of the first image. Alternatively, if the screen refresh rate is less than or equal to the refresh rate threshold, the GPU performs blur processing on the first layer of the second image according to the blur parameters to obtain the blurred layer of the second image; and updates the blurred layer of the second image to the blur buffer.
4. The method of claim 3, wherein, The second interface also includes a third image, where the first image is the a-th frame image, the second image is the a+1-th frame image, and the third image is the b-th frame image, where a is less than b, and both a and b are positive integers; the first image to the b-1-th frame images are all composited by HWC using the blur layer of the first image; After not updating the blur layer of the first image in the blur buffer, the method further includes: Obtain multiple layers of the third image, wherein the multiple layers of the third image include the first layer of the third image and the second layer of the third image; If the size or position of the second layer of the third image is different from that of the second layer of the first image, and the screen refresh rate is greater than the refresh rate threshold, determine whether the number of times the blurred layer of the first image is used has reached the first count threshold. If the first threshold number of times is reached, the GPU performs blur processing on the first layer of the third image according to the blur parameters to obtain the blurred layer of the third image; and updates the blurred layer of the third image to the blur buffer. The HWC composites the blurred layer, the first layer, and the second layer of the third image to obtain the third image; The third image is displayed.
5. The method of claim 2, wherein, The second interface also includes a fourth image, where the first image is the c-th frame image, and the fourth image is the d-th frame image, where c is less than d, and both c and d are positive integers; the second layer in the (c+1)-1 to (d-1)-th frames is the same size and position as the second layer in the first image; the first layer in the (c+1)-1 to (d-1)-th frames is the same as the first layer in the first image; and the (c+1)-1 to (d-1)-th frames are all composited by HWC using the blurred layer of the first image. After displaying the first image, the method further includes: Obtain multiple layers of the fourth image, wherein the multiple layers of the fourth image include the first layer of the fourth image and the second layer of the fourth image; When the second layer of the fourth image and the second layer of the first image have the same size or position, and the first layer of the fourth image and the first layer of the first image have different appearances, the GPU performs blur processing on the first layer of the fourth image according to the blur parameters to obtain the blurred layer of the fourth image; and updates the blurred layer of the fourth image to the blur buffer.
6. The method of claim 2, wherein, The second interface also includes a fifth image, which is the next frame image after the first image. After displaying the first image, the interface further includes: Obtain multiple layers of the fifth image, wherein the multiple layers of the fifth image include the first layer of the fifth image and the second layer of the fifth image; When the second layer of the fifth image and the second layer of the first image have the same size or position, and the first layer of the fifth image and the first layer of the first image have different images, the HWC composites the blurred layer of the first image, the first layer of the fifth image, and the second layer of the fifth image to obtain the fifth image; the blurred buffer stores the blurred layer of the first image; The fifth image is displayed.
7. The method of claim 6, wherein, The second interface also includes a sixth image, where the first image is the e-th frame image, and the sixth image is the f-th frame image, where e is less than f, and both e and f are positive integers; the first image to the (f-1)-th frame images are all composited by HWC using the blur layer of the first image; after displaying the fifth image, the interface further includes: Obtain multiple layers of the sixth image, wherein the multiple layers of the sixth image include the first layer of the sixth image and the second layer of the sixth image; If the second layer of the sixth image and the second layer of the first image have the same size or position, and the first layer of the sixth image and the first layer of the first image have different images, determine whether the number of times the blurred layer of the first image is used has reached the second threshold. If the second threshold number of times is reached, the GPU performs blur processing on the first layer of the sixth image according to the blur parameters to obtain the blurred layer of the sixth image; and updates the blurred layer of the sixth image to the blur buffer. The HWC composites the blurred layer, the first layer, and the second layer of the sixth image to obtain the sixth image; The sixth image is displayed.
8. The method according to any one of claims 1 to 7, characterized in that, Before displaying the second interface, the following is also included: In response to the first operation, multiple layers of the seventh image are obtained; the image of the first interface is the g-th frame image, and the seventh image is the g+1-th frame image, where g is a positive integer; the multiple layers of the seventh image include the first layer of the seventh image and the second layer of the seventh image; Under the condition of satisfying the first preset condition, the blur buffer is created; at the same time, the blur parameter is masked, and the first layer and the second layer of the seventh image are composited using the HWC to obtain the seventh image; wherein, during the process of compositing the seventh image, the blur buffer does not include the blur layer; the first preset condition includes: the second layer carries the blur parameter, the second layer is the preset layer, the number of the second layer is 1, and the first layer is not a safe layer.
9. The method according to any one of claims 1 to 8, characterized in that, After receiving the first operation in the first interface, the process also includes: According to the first operation, a third interface is displayed; the third interface includes the first layer, the second layer, and the blurred layer corresponding to the second layer; the blurred layer is obtained from the temporary buffer of the GPU; when the multiple layers of the third interface meet the second preset conditions, the third interface is synthesized by the GPU from the multiple layers of the third interface. The second preset condition includes one or more of the following: the second layer carries the blur parameter, and the number of second layers is greater than 1; or, the second layer carries the blur parameter, and the preset layer does not include the second layer; or, the second layer carries the blur parameter, and the first layer is a safe layer.
10. An image compositing method characterized by comprising: Applied to an electronic device, the electronic device including a hardware hybrid renderer (HWC) and a graphics processing unit (GPU), the method includes: Obtain multiple layers of the first interface; the multiple layers of the first interface include the first layer, but do not include the blurred layer, and the size of the first layer is the same as the screen size of the electronic device; The HWC synthesizes multiple layers of the first interface to obtain the first interface; Display the first interface; Receive the first operation in the first interface; Obtain multiple layers from the second interface; the multiple layers of the second interface include the first layer and the second layer. Determine whether multiple layers of the second interface meet the first preset condition; the first preset condition includes that the second layer carries the blur parameter, the second layer is a preset layer, the number of the second layer is 1, and the first layer is not a safe layer; the preset layer includes any one of the following: volume bar layer, message pop-up layer, or sidebar layer; If the first preset condition is met, the blurred layer corresponding to the second layer is obtained from the blurred buffer; The HWC composites the first layer of the second interface, the second layer of the second interface, and the blurred layer of the second interface to obtain the second interface; the blurred layer is located above the first layer, and the second layer is located above the blurred layer. Alternatively, if the first preset condition is not met and the second preset condition is met, the GPU performs blurring processing on the first layer of the second interface to obtain the blurred layer; the blurred layer is stored in a temporary buffer; the second preset condition includes one or more of the following: the second layer carries the blur parameter, and the number of second layers is greater than 1; or, the second layer carries the blur parameter, and the preset layer does not include the second layer; or, the second layer carries the blur parameter, and the first layer is a safe layer; The GPU composites the first layer of the second interface, the second layer of the second interface, and the blurred layer of the second interface to obtain the second interface; The second interface is displayed.
11. The method of claim 10, wherein, The second interface includes a first image and a second image; the method includes: After obtaining the first layer and the second layer of the first image, the GPU performs blur processing on the first layer of the first image according to the blur parameters to obtain the blurred layer of the first image; the blurred layer of the first image is stored in the blur buffer. Obtain the blurred layer of the first image from the blurred buffer; The HWC composites the first layer, the second layer, and the blurred layer of the first image to obtain the first image; After obtaining the first layer of the second image and the second layer of the second image, the blurred layer of the first image is obtained from the blurred buffer; The HWC composites the first layer of the second image, the second layer of the second image, and the blurred layer of the first image to obtain the second image.
12. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-11.
13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.
14. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being configured to run a computer program or instructions to perform the method as described in any one of claims 1-11.
15. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-11.
