Image processing method and apparatus

Abstract

Embodiments of the present application relate to the technical field of terminals, and provide an image processing method and apparatus. The method comprises: running a first application and a second application; and compositing a first layer corresponding to a first window and a second layer corresponding to a second window to obtain a first image, wherein the second window comprises a first area and a second area, the first area is a rounded corner area in the second window corresponding to the position of the first window, and the second area is the area in the second window other than the first area. In this way, instead of using a GPU to composite a rounded corner effect for the first window, an electronic device reserves the rounded corner area in the second window, and then composites the layers respectively corresponding to the two windows, so that when the layers are composited, the content in the first window can be displayed in the rounded corner area, thereby achieving the effect of reducing GPU power consumption.

Description

Image processing methods and apparatus Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and apparatus. Background Technology

[0002] Mobile phones, tablets, and other electronic devices typically support multitasking. Multitasking refers to running multiple applications in the foreground, allowing users to use multiple applications simultaneously. A typical multitasking scenario involves an electronic device displaying an application interface while simultaneously showing a video in a pop-up window, allowing users to watch a video while using another application. For ease of explanation, this typical multitasking scenario will be referred to as the "video pop-up scenario" below.

[0003] In the aforementioned video window scenario, the outer contour of the video window is usually a rounded rectangle. Correspondingly, during the processing of each frame of the image, the electronic device needs to process the image to obtain the rounded corner effect of the rounded rectangle. The process of compositing the rounded corner effect greatly increases the possibility of the electronic device overheating, affecting the performance of the electronic device.

[0004] Summary of the Invention

[0005] This application provides an image processing method and apparatus, applied in the field of terminal technology, enabling electronic devices to reduce the possibility of overheating and improve the performance of electronic devices in multi-task operation scenarios including rounded corner windows (such as the video window scenario mentioned above) by not using GPU to synthesize rounded corner effects.

[0006] In a first aspect, embodiments of this application propose an image processing method and apparatus. The method includes: running a first application and a second application, the first application corresponding to a first window, the first window being a non-full-screen window, and the second application corresponding to a second window, the second window being a full-screen window, the second application being a game application; compositing a first layer corresponding to the first window and a second layer corresponding to the second window to obtain a first image, wherein the second window includes a first region and a second region, the first region being a rounded corner region in the second window corresponding to the position of the first window, and the second region being the region in the second window excluding the first region; and displaying the first image.

[0007] The first application is a video application (or video player), the second application is a game application, the first layer is a video window layer, the second layer is a processed game layer, and the first image can be the target image frame 1.

[0008] Electronic devices can avoid using the GPU to composite the rounded corners of the first window. Instead, they can reserve a rounded corner area in the second window and then composite the layers corresponding to the two windows respectively. This allows the content in the first window to be displayed in the rounded corner area after the layers are composited, thus saving GPU power consumption.

[0009] Running the first and second applications includes: first, launching the first application; displaying a first window related to the first application in response to an operation on the first application; and then launching the second application. As shown in Figure 1, when the first application is a video application, the electronic device can display the first window in response to the user swiping up from the bottom of the first application. Then, the user can launch the second application by clicking its icon on the desktop application.

[0010] In one possible implementation, the method further includes: obtaining the first layer and obtaining the second layer. The layers need to be processed and obtained before layer compositing to ensure that the processed layers can present the desired effect.

[0011] In one possible implementation, obtaining the second layer includes: performing pixel rendering on a second region within a second window to obtain the second layer.

[0012] It is understandable that during the pixel rendering process of the second region in the second window, the electronic device does not perform pixel rendering of the first region in the second window, or it can be understood that the electronic device only performs pixel rendering of the first region and the second region in the second region.

[0013] Electronic devices can reserve rounded corner areas (or "holes") in the second window by rendering only the second area pixel by pixel. This allows the content of the first window to be displayed within these rounded corner areas, ensuring that the first window is displayed as a rounded rectangle on the interface. Simultaneously, rendering only the second area pixel by pixel also saves device power consumption during the pixel rendering process.

[0014] In one possible implementation, obtaining the second layer includes: performing pixel rendering on the first region and the second region in the second window to obtain the pixel-rendered second window, and using the position of the first window to make the first region in the pixel-rendered second window transparent to obtain the second layer.

[0015] Electronic devices can reserve rounded corner areas (or "holes") in the second window by rendering only the first and second regions pixel-wise and by making the first region transparent. This allows the content in the first window to be displayed within these rounded corner areas. Simultaneously, this transparency process does not affect the pixel rendering steps, thus improving system stability.

[0016] In one possible implementation, obtaining the second layer includes: obtaining the system state of the electronic device; and obtaining the second layer when it is detected that the system state is not in a power-saving state.

[0017] Electronic devices can process the second layer in non-power-saving mode to reduce the impact of higher power consumption when processing the second layer in power-saving mode on the user's use of other functions.

[0018] In one possible implementation, obtaining the second layer includes: obtaining the system state of the electronic device; if it is detected that the system state is not in the device rotation state, obtaining the second layer, wherein the device rotation state includes: the state where the device switches from landscape to portrait, and / or the state where the device switches from portrait to landscape.

[0019] It is understandable that when the device is rotated, there is a difference between the actual position of the first window and the reserved rounded corner area in the second window, which causes the content in the first window to be unable to be fully displayed in the rounded corner area. Therefore, the electronic device can process the second layer when it detects that the device is not in a rotating state, so as to reduce the display content disorder.

[0020] In one possible implementation, obtaining the second layer includes: obtaining the motion state of the first window; if the first window is detected to be not in motion, obtaining the second layer, wherein the first window being in motion includes: the size of the first window changing, and / or the position of the first window changing.

[0021] It is understandable that when the first window is in motion, there is a difference between the actual position of the first window and the reserved rounded corner area in the second window, which causes the content in the first window to be unable to be fully displayed in the rounded corner area. Therefore, the electronic device can process the second layer when it detects that the first window is not in motion, so as to reduce the display content disorder.

[0022] In one possible implementation, the electronic device includes an image compositor, an application-aware component, and a hardware compositor (HWC). After running the first and second applications, the method further includes: when the image compositor detects the presence of a rounded corner window, the image compositor sends the position of the first window and the identifier of the second application to the application-aware component; obtaining a second layer, including: the application-aware component obtaining the second layer based on the position of the first window; the method further includes: the application-aware component sending the second layer and a first message to the image compositor; in response to the first message, the image compositor does not invoke the image processor (GPU) to composite the rounded corner effect of the first window; compositing the first layer corresponding to the first window and the second layer corresponding to the second window, including: the image compositor invoking the HWC to composite the first layer and the second layer.

[0023] In the second application scenario, which is a game application, electronic devices can achieve rounded corner effects by calling the image synthesizer, application awareness, and HWC to perform layer processing (such as pixel rendering) and by adjusting the layer processing. This replaces the GPU in generating rounded corner effects, thereby saving GPU power consumption.

[0024] In one possible implementation, the electronic device further includes: a GUE service, wherein the image synthesizer sends the position of the first window and the identifier of the second application to the application perception, including: the image synthesizer sending the position of the first window and the identifier of the second application to the GUE service; and when the GUE service detects that the system state of the electronic device and / or the running state of the second application meet preset conditions, the GUE service sends the position of the first window and the identifier of the second application to the application perception.

[0025] The GUE service can detect the system status and / or the running status of the application, and then determine whether the second layer can be processed. If the GUE service determines that the preset switch can be turned on based on the system status and / or the running status of the application, it will notify the application to perceive and process the layer.

[0026] In one possible implementation, the image synthesizer sends the position of the first window and the identifier of the second application to the application perception, including: when the image synthesizer detects a preset layer, the image synthesizer sends the position of the first window and the identifier of the second application to the application perception, the preset layer including: the layer corresponding to the game application.

[0027] Electronic devices can identify the scene by judging the preset layers and determine how to process the layers, thus enriching the application scenarios of image processing methods and improving the stability and accuracy of image processing methods.

[0028] In one possible implementation, the process of compositing the first and second layers passes through the HWC of the electronic device to achieve power savings for the GPU.

[0029] In one possible implementation, during the compositing of the first and second layers, the second layer is overlaid on top of the first layer, allowing the content of the first layer to be displayed in the rounded corner area of ​​the second layer.

[0030] In one possible implementation, after obtaining the first image, the method further includes: in response to the operation of exiting the second window, running a first application and a third application, the third application corresponding to the third window, the third window being a full-screen window, and the third application being a desktop application; compositing the fourth layer corresponding to the first window and the third layer corresponding to the third window to obtain the second image, wherein the third window includes: a third region and a fourth region, the third region being a rounded corner region in the third window corresponding to the position of the first window, and the fourth region being the region in the third window excluding the third region; and displaying the second image.

[0031] The third application can be a desktop application. Electronic devices can avoid using the GPU to composite the rounded corners of the first window. Instead, they can reserve a rounded corner area in the third window and then composite the layers corresponding to the two windows respectively. This allows the content in the first window to be displayed in the rounded corner area after the layers are composited, thus saving GPU power consumption.

[0032] In one possible implementation, the method further includes: obtaining a fourth layer and obtaining a third layer; obtaining the third layer includes: drawing a transparent rounded rectangle in the third region and drawing the first content to be displayed in the fourth region, thus obtaining the third layer. Electronic devices can achieve a rounded corner effect by drawing a transparent rounded rectangle in the third region.

[0033] In one possible implementation, the electronic device includes: an image compositor, a rendering module, and an HWC; after running the first and third applications, the method further includes: when the image compositor detects the presence of a rounded corner window, the image compositor sends a second message and the position of the first window to the rendering module; obtaining a third layer includes: in response to the second message, the rendering module obtains the third layer using the position of the first window; the method further includes: the rendering module sends a third message and the third layer to the image compositor; in response to the third message, the image compositor does not call the image processor GPU to composite the rounded corner effect of the first window; compositing the fourth layer corresponding to the first window and the third layer corresponding to the third window includes: the image compositor calls the HWC to composite the fourth layer and the third layer; wherein, the second content to be displayed in the fourth layer is drawn by the rendering module or refreshed by the image compositor.

[0034] In the scenario where the third application is a desktop application, electronic devices can achieve rounded corner effects by calling the image synthesizer, drawing module, and HWC to perform layer processing (such as drawing content in the layer), and by adjusting the layer processing process, thereby replacing the GPU in generating rounded corner effects and saving GPU power consumption.

[0035] In one possible implementation, the image compositor sends a second message and the position of the first window to the rendering module, including: if the image compositor does not detect a preset layer, the image compositor sends a second message and the position of the first window to the rendering module, wherein the preset layer includes: the layer corresponding to the game application.

[0036] Electronic devices can identify the scene by judging the preset layers and determine how to process the layers, thus enriching the application scenarios of image processing methods and improving the stability and accuracy of image processing methods.

[0037] In one possible implementation, pixel rendering is performed on a second region within a first window to obtain a second layer, including: pixel rendering of the second region within the first window if the identifier of the second application belongs to the first list. Alternatively, pixel rendering is performed on both the first and second regions within the second window, including: pixel rendering of both the first and second regions within the second window if the identifier of the second application does not belong to the first list.

[0038] The first list can be a whitelist 2. Electronic devices can use the list to filter and perform different pixel rendering processes for different game applications to improve the stability of the entire image processing process.

[0039] In one possible implementation, obtaining the second layer includes: obtaining the second layer when the system state of the electronic device and / or the operating state of the second application meet preset conditions.

[0040] In one possible implementation, the system state includes one or more of the following: power saving state, device rotation state, first window motion state, or multi-game state, etc., and the running state of the second application includes: frame interpolation state and / or non-full-screen state. Detecting that the system state of the electronic device and / or the running state of the second application meet preset conditions includes: detecting that the electronic device does not meet all states in the system state, and / or detecting that the second application does not meet all states in the running state of the second application.

[0041] It is understood that electronic devices can improve the accuracy of image processing methods by detecting the system state and / or the operating state of the second application, that is, to execute the image processing method provided in the embodiments of this application in the most suitable state of the electronic device.

[0042] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip, or a chip system within an electronic device. The image processing apparatus may include a display unit and a processing unit. When the image processing apparatus is an electronic device, the display unit may be a display screen. The display unit is used to perform display steps to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is a chip or a chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0043] Specifically, the processing unit is used to run a first application and a second application. The first application corresponds to a first window, which is a non-full-screen window. The second application corresponds to a second window, which is a full-screen window. The second application is a game application. The processing unit is also used to composite the first layer corresponding to the first window and the second layer corresponding to the second window to obtain a first image. The second window includes a first region and a second region. The first region is the rounded corner region in the second window corresponding to the position of the first window, and the second region is the region in the second window excluding the first region. The display unit is used to display the first image.

[0044] Thirdly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect or any possible implementation thereof.

[0046] Fifthly, embodiments of this application provide a computer program product including a computer program. When the computer program product includes computer program code, when the computer program code is run on an electronic device, it causes the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0047] Sixthly, this application provides a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0048] In one possible implementation, the chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip system, such as a register or cache, or it can be a storage unit of the chip system itself (e.g., read-only memory, random access memory, etc.).

[0049] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0050] Figure 1 is a schematic diagram of a scenario provided by an embodiment of this application;

[0051] Figure 2 is a schematic diagram of another scenario provided by an embodiment of this application;

[0052] Figure 3 is a schematic diagram of an image processing procedure provided in an embodiment of this application;

[0053] Figure 4 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0054] Figure 5 is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0055] Figure 6 is a schematic diagram of the module interaction of an image processing method provided in an embodiment of this application;

[0056] Figure 7 is a schematic diagram of another image processing procedure provided in an embodiment of this application;

[0057] Figure 8 is a schematic diagram of an interface provided in an embodiment of this application;

[0058] Figure 9 is a schematic diagram of the module interaction of another image processing method provided in an embodiment of this application;

[0059] Figure 10 is a schematic diagram of another interface provided in an embodiment of this application;

[0060] Figure 11 is a schematic diagram of the module interaction of another image processing method provided in an embodiment of this application;

[0061] Figure 12 is a schematic diagram of the module interaction of another image processing method provided in the embodiment of this application;

[0062] Figure 13 is a flowchart illustrating an image processing method provided in an embodiment of this application;

[0063] Figure 14 is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0064] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0065] 1. Rounded corner window

[0066] A rounded-corner window can be understood as a window with a rounded rectangle outline. For example, a rounded-corner window can be a video window, an application floating window, or a mini window. Among them, the video window can display video content, while the application floating window or mini window can display any interface of the application, and the application floating window and mini window can be switched based on user operation.

[0067] A diagram of a rounded window can be found in Figure 1 or Figure 2, and will not be repeated here.

[0068] 2. Rounded corner layer

[0069] A rounded corner layer can be understood as a layer with a rounded corner attribute. The rounded corner attribute indicates that the content displayed in the rounded corner layer will have a rounded corner effect in the display interface. This rounded corner effect is not directly displayed in the rounded corner layer; for example, the rounded corner layer can be a rectangle.

[0070] When an electronic device launches a rounded corner window, a rounded corner layer corresponding to the rounded corner window is created. The rounded corner layer can be used to draw (or refresh) the content displayed in the rounded corner window.

[0071] Electronic devices can determine the presence of rounded corner windows by recognizing properties such as the rounded corners of layers. For example, if image frame 1 includes N layers (e.g., N layers involved in compositing), and a layer M containing rounded corners is identified among these N layers, it is determined that a certain layer is a rounded corner layer, and a rounded corner window exists in image frame 1. Alternatively, if no layer containing rounded corners is identified among these N layers, it is determined that no rounded corner layer exists, and a rounded corner window does not exist in image frame 1.

[0072] Optionally, the electronic device can also record a whitelist 1, which includes identifiers of layers that can be used to draw specific content, such as layer names. Specific content includes content that may be displayed in a rounded corner window. For example, specific content includes video content or email interface content. The electronic device can match the layers included in an image frame with whitelist 1 to determine whether a rounded corner window exists. For example, if image frame 2 includes Q layers, and if the layer identifiers corresponding to the Q layers are found to contain layer identifiers included in whitelist 1, it is determined that a rounded corner layer exists, and a rounded rectangle exists in image frame 2. If the layer identifiers corresponding to the Q layers are not found to contain layer identifiers included in whitelist 1, it is determined that a rounded corner layer does not exist, and a rounded rectangle does not exist in image frame 2.

[0073] It is understood that the embodiments of this application do not limit the method for detecting the existence of rounded corner windows.

[0074] 3. Other terms

[0075] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0076] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0077] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0078] 4. Electronic equipment

[0079] The electronic devices in this application embodiment may include handheld devices with image processing capabilities, in-vehicle devices, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or terminal devices in future evolved public land mobile networks (PLMNs), etc., and this application embodiment is not limited to these.

[0080] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Furthermore, in this embodiment, the electronic device can also be a terminal device in an Internet of Things (IoT) system.

[0081] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0082] This application provides an image processing method that can be applied to multi-task operation scenarios including rounded corner windows, and processes the image to obtain image frames with rounded corner effects.

[0083] The following description, with reference to the embodiments corresponding to Figures 1 and 2, illustrates the content contained in the rounded corner window and the display process of the rounded corner window. In this embodiment, a mobile phone is used as an example for illustration, and this example does not constitute a limitation on the embodiments of this application.

[0084] The rounded corner window can be a small video window (see the description in Figure 1) or an application floating window (see the description in Figure 2).

[0085] One implementation illustrates the process of launching a video window on an electronic device.

[0086] Figure 1A shows the application interface of the video player, where video 1011 is playing.

[0087] In response to a user's swipe gesture from the bottom of Figure 1A upwards (as indicated by the arrow in Figure 1A), the electronic device can display the interface shown in Figure 1B.

[0088] Optionally, a button 1 for triggering the display of the small window can also be displayed in the video 1011 of Figure 1A. In response to the user's click operation on button 1, the electronic device can display interface 1 containing the video small window 1021. At this time, other areas of interface 1 besides the video small window 1021 can display content related to the video player. It is understood that since button 1 is not shown in Figure 1A and interface 2 is not shown, the embodiments of this application do not limit the method of triggering the display of the video small window.

[0089] Figure 1B can be the desktop of an electronic device, and Figure 1B includes: a video window 1021 and a game application icon 1031. Video 1011 is playing in the video window 1021. The video window 1021 is a rounded window.

[0090] In response to a user's click on icon 1031, the electronic device can display the interface shown in Figure 1C.

[0091] Figure 1C may include: the game application's startup screen and a video window 1021.

[0092] The image processing method provided in this application embodiment can be used to process and obtain the target image frame shown in FIG1C. The target image frame includes: a video window 1021 with rounded corners and any interface in a game application.

[0093] Another implementation illustrates the process of launching an application's floating window on an electronic device.

[0094] Figure 2A shows the startup interface of a game application.

[0095] In response to a user's swipe operation from the right edge of Figure 2A to the left (in the direction indicated by the arrow in Figure 2A), the electronic device can display the interface shown in Figure 2B.

[0096] Figure 2B may include: the game application's launch screen, and window 2021. Window 2021 may include application icons for multiple shortcut applications, such as the icon 2022 for Favorites, the icon 2023 for Email, the icon 2024 for Notes, etc.

[0097] In response to a user's click on icon 2023, the electronic device can display the interface shown in Figure 2C.

[0098] Figure 2C may include: the game application's launch screen, and the application floating window 2025. The application floating window 2025 may display: the email application interface. The application floating window 2025 is a rounded-corner window.

[0099] The image processing method provided in this application embodiment can be used to process and obtain the target image frame shown in FIG2C. The target image frame includes: an application floating window 2025 with rounded corners and any interface in a game application.

[0100] Referring to the descriptions in Figures 1 and 2, the electronic device can launch the rounded corner window first and then launch the game application, or it can launch the game application first and then launch the game window. This application embodiment does not limit the order of launching the rounded corner window and launching the game application.

[0101] Figures 1 and 2 above only show two typical rounded-corner windows, and are not limited to these in practice. The following explanation will primarily use a rounded-corner window as an example in a video window scenario.

[0102] The rounded corner effect of a rounded window is achieved by the GPU in the electronic device through compositing (such as performing rounded corner clipping). Specifically, the electronic device can obtain (e.g., through drawing, refreshing, etc.) the content of multiple layers, including the rounded corner window layer (hereinafter referred to as the rounded corner layer). The GPU performs rounded corner clipping on the rounded corner layer, and the clipped rounded corner layer is overlaid with other layers to obtain an image frame.

[0103] Multiple layers include, as shown in Figure 3: a video window layer 301 and a game layer 302. The electronic device can refresh video content in the video window layer 301 and draw the game interface (such as any interface in a game application, like the game application's launch screen) in the game layer 302. It can be understood that the electronic device decodes the video file using a codec component (such as mediacodec), obtains video frames, and then refreshes them into the video window layer 301, thus obtaining the content in the video window layer 301 without needing to draw it.

[0104] The electronic device can use the GPU to composite the video window layer 301 to obtain a video window layer 303 with rounded corners. Then, the video window layer 303 with rounded corners and the game layer containing the game interface are superimposed from top to bottom, with the video window layer 303 with rounded corners superimposed on top, to obtain image frame 304. Image frame 304 includes the video window with rounded corners.

[0105] In the above scheme, the electronic device uses the GPU to complete the rounded corner effect, which will increase the GPU load during image processing, greatly increasing the possibility of the electronic device overheating and affecting its performance.

[0106] In view of this, embodiments of this application provide an image processing method that enables an electronic device to, after detecting a game layer and a rounded corner layer, reserve the position of the rounded corner window (or understand it as cutting a hole) by not rendering pixels in the area where the rounded corner window is located in the game layer, or by making the area where the rounded corner window is located in the game layer transparent, and then overlay the rounded corner layer and the game layer with the reserved rounded corner window to obtain an image frame with a rounded corner window.

[0107] In this way, electronic devices can achieve rounded corner effects by either not rendering the area containing the rounded corner window in the game layer as pixels, or by making that area transparent, thus replacing GPU-composited rounded corner effects. Since rounded corner effects don't require GPU compositing, the load on the GPU during image processing is reduced, lowering the likelihood of overheating and improving the performance of electronic devices, such as enabling smooth multitasking.

[0108] To better understand the embodiments of this application, the structure of the electronic device according to the embodiments of this application will be described below. For example, Figure 4 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0109] As shown in Figure 4, taking a mobile phone as an example, the electronic device may include a processor 210, an external memory interface 220, an internal memory (RAM) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0110] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] The processor 210 may include one or more processing units, such as an application processor (AP), a modem processor, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0112] The processor 210 can implement steps in the image processing method such as not rendering pixels in the area where the rounded corner window is located, or making the area where the rounded corner window is located transparent, as well as steps such as overlaying the game layer with reserved rounded corner window and the rounded corner layer to obtain an image frame with rounded corner window.

[0113] The charging management module 240 receives charging input from the charger. The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, display screen 294, camera 293, and wireless communication module 260, etc. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance).

[0114] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor. Mobile communication module 250 can provide wireless communication solutions for mobile phones, including 2G / 3G / 4G / 5G. Wireless communication module 260 can provide wireless communication solutions for mobile phones, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0115] The mobile phone can achieve display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0116] Mobile phones can achieve camera functions through the camera 293, ISP, video codec, GPU, display 294, application processor (AP), neural network processor (NPU), etc.

[0117] The display 294 can display any of the interfaces described in Figures 1-2, or any of the interfaces described in Figures 8 and 10.

[0118] Mobile phones can perform audio functions, such as music playback and recording, through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, and an application processor.

[0119] The software system of the aforementioned electronic devices can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture.

[0120] This application embodiment uses a layered architecture of Android. TM Taking a system as an example, this section illustrates the software structure of an electronic device. A layered architecture divides the software system of an electronic device into several layers, each with a clear role and function, and the layers communicate with each other through software interfaces.

[0121] For example, Figure 5 is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. As shown in Figure 5, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces.

[0122] In some embodiments, the Android system is divided into multiple layers, from top to bottom, which are the hardware and software architecture of the electronic device and may include an application (APP) layer, an application framework layer, a native layer, a kernel layer, and a hardware layer.

[0123] The application layer can include one or more of the following: game applications, video players, launcher applications, and email applications. The phone can run multiple of these applications in the foreground, and one of these applications can include a rounded-corner window.

[0124] Taking the scenario shown in Figure 1 above (a specific video window scenario) as an example, the mobile phone can run a game application and a video player in the foreground, and the video player includes a rounded corner window.

[0125] Taking the scenario shown in Figure 2 above as an example, the mobile phone can run a game application and an email application in the foreground, and the email application includes a rounded corner window.

[0126] The application framework layer provides the application programming interface (API) and programming framework for the application layer.

[0127] The application framework layer can include various system services, such as the window manager service (WMS) and the display manager service (DMS).

[0128] WMS can be used for window management, window animation management, surface management, and as a relay station for the input system. DMS can be used for refreshing application views, managing framebuffers, and as a pathway for switching between displays in dual-screen setups.

[0129] The application framework layer may include: view system API, graphics system API, and game kit.

[0130] The View System API can be used for application layout calculations and the execution of drawing commands. For example, layout calculations include calculating the positions of various icons on the desktop. Furthermore, the View System API includes classes such as View, ViewGroup, and SurfaceView.

[0131] The Graphics System API can be used to set relevant properties of graphics. Furthermore, the Graphics System API includes classes such as canvas, render node, and paint.

[0132] GameKit can be used to provide a usable interface for game applications.

[0133] The native layer provides various services to the upper layers (such as the application framework layer). For example, the native layer includes: a drawing module (libhwui), a capability library for the Skia (an open-source 2D graphics library) API (libskia) (or simply a graphics capability library), an image synthesizer (surfaceflinger), an embedded graphics library (libEGL), an API support module that provides the Open Graphics Library (OpenGL) (libGLES_XX) (or simply a graphics support module), GUE services, a layer management module (libgui), and game awareness (gamesaware), etc.

[0134] The term "surfaceflinger" can also be translated as "surface thrower," "surface drawing module," or "image compositing service," but this application does not specifically limit it to these terms.

[0135] In this context, libhwui can be used for image rendering in the second scenario. The second scenario can be understood as an electronic device running both a desktop application and a video player simultaneously in the foreground. In this second scenario, the electronic device can achieve a rounded corner effect by creating a cutout in the desktop layer and overlaying a rounded corner layer over the cutout area.

[0136] It's understandable that each application can complete its drawing through its own rendering thread. The rendering thread uses values ​​from libhwui to perform the drawing. In other words, each application has its own libhwui library for implementing its specific drawing functionality; for example, desktop applications use libhwui 1.

[0137] In the following text, for the sake of simplicity, the actions of renderthread will be represented as the actions of libhwui, such as renderthread drawing being represented as libhwui drawing.

[0138] libhwui is an important component of the Android system, short for Hardware Accelerated Rendering Engine for UI. It is a GPU-accelerated 2D graphics engine designed to provide efficient, stable, and high-quality 2D graphics rendering capabilities, supporting the Android system's user interface (UI) experience.

[0139] libskia can be used by libhwui to submit drawing commands to the GPU for rendering. libGLES_XX can be used to convert the drawing commands submitted by libskia into GLES format for GPU execution. libgui can be used for layer and layer attribute rotation; for example, libhwui uses libgui to submit layers and layer attributes to SurfaceFlinger.

[0140] Gamesaware can be used to filter schemes based on the application's identifier. For example, if Gamesaware detects that the application's identifier is in whitelist 2, it will execute scheme one; or if it detects that the application's identifier is not in whitelist 2, it will execute scheme two.

[0141] Option 1 could include: gamesaware processes the game layer by not rendering the area where the rounded corner window is located, thus obtaining a game layer with reserved rounded corner windows, and achieving the compositing of rounded corner effects.

[0142] Option 2 could include: gamessaware processing the game layer by making the area containing the rounded corner window transparent, obtaining the game layer with the reserved rounded corner window, and compositing the rounded corner effect.

[0143] It should be noted that gamesaware can be a module bound to a game application; that is, game application 1 can correspond to gamesaware1, and game application 2 can correspond to gamesaware2. To implement the image processing method provided in this application embodiment, the scheme selection process, as well as the implementation processes of scheme one and scheme two, need to be pre-set in the gamesaware corresponding to the game application.

[0144] The GUE service can be used to determine whether to enable or disable a preset switch based on system and / or game status.

[0145] The preset switch can be used to start the image processing process in the first scenario. The first scenario can be understood as a scenario in which the electronic device is running a game application and a rounded corner window at the same time in the foreground (as described in Figure 1 or Figure 2).

[0146] The GUE service can be used to send the position information of rounded windows and the identifier of the game application (such as the package name of the game application) to gamesaware.

[0147] It is understandable that GUE services can be understood as a game status control center, that is, determining whether to turn on a preset switch based on the system status and / or game status.

[0148] SurfaceFlinger can be used for refresh rate control, image compositing control, and more. For example, SurfaceFlinger can make decisions to assign compositing tasks to a hardware composer (HWC) or a GPU.

[0149] SurfaceFlinger can be used to monitor rounded corner windows, such as to detect whether a rounded corner window exists.

[0150] For example, when SurfaceFlinger detects the presence of a rounded window, such as a video window, SurfaceFlinger can query the GUE service via message 1 to check whether the preset switch should be enabled.

[0151] Optionally, if SurfaceFlinger detects the existence of a rounded corner window and the rounded corner window remains stable, SurfaceFlinger can query the GUE service via message 1 to determine whether to enable the preset switch. If a rounded corner window is detected but the rounded corner window is unstable (e.g., its position or size is changing), or if a rounded corner window is detected but not present, SurfaceFlinger invokes the GPU to perform the compositing of the rounded corner effect.

[0152] SurfaceFlinger can be used to send the position information of rounded windows, such as the position information of video windows, to GUE services.

[0153] The kernel layer includes drivers that drive the hardware, such as display drivers and GPU drivers. Display drivers can drive the display screen to show images, such as driving the monitor to display image frames including rounded corner windows. GPU drivers can drive the GPU to work, such as driving the GPU to perform image compositing.

[0154] It is understood that the embodiments of this application do not specifically limit the software layers involved in the software architecture, the modules contained in the software layers, and the functions of the modules.

[0155] Referring to the description of the software architecture in Figure 5, the technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems will be described in detail below with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0156] The present application provides an image processing method that enables an electronic device to perform image processing in a first scene, as described in Figures 6-11; or the electronic device may also detect a first scene or a second scene to determine whether to perform image processing in the first scene or the second scene, as described in Figure 12.

[0157] The image processing process in the first scenario is illustrated below with reference to the embodiments corresponding to Figures 6-11. Specifically, when the rounded corner window is activated, the image processing process can be referred to the descriptions in Figures 6-7; when the position or size of the rounded corner window is adjusted, the image processing process can be referred to the descriptions in Figures 8-9; and when the rounded corner window is closed, the image processing process can be referred to the descriptions in Figures 10-11.

[0158] Figure 6 is a schematic diagram of the module interaction of an image processing method provided in an embodiment of this application. In the embodiment corresponding to Figure 6, a rounded corner window is used as a video window and a rounded corner layer is used as a video window layer for illustrative purposes. This example does not constitute a limitation on the embodiments of this application.

[0159] As shown in Figure 6, the image processing method may include the following steps:

[0160] S601. In response to the power-on or restart of an electronic device, SurfaceFlinger starts a listener for rounded corner windows.

[0161] For example, SurfaceFlinger can initialize after an electronic device is powered on or restarted. During SurfaceFlinger initialization, a listener for rounded corner windows can be started. After starting the rounded corner window listener, SurfaceFlinger can detect the opening and closing of rounded corner windows.

[0162] In some embodiments, SurfaceFlinger can monitor rounded corner windows by starting a listening service.

[0163] Understandably, when SurfaceFlinger starts listening for rounded corner windows, when the electronic device detects event 1, SurfaceFlinger can listen for the presence of rounded corner windows, as described in S602.

[0164] S602. In response to event 1, the video player sends a request to WMS to create a video window layer.

[0165] Event 1 can be used to trigger the launch of a rounded corner window. For example, Event 1 can be a user swiping action from the bottom of the video playback interface upwards, as described in Figure 1.

[0166] The video window layer creation request can be used to request the creation of a video window layer for playing video content.

[0167] For example, if the video player detects event 1, it can determine that the video needs to be played in a video window and then sends a request to WMS to create a video window layer.

[0168] S603, In response to event 2, the game application sends a request to WMS to create a game layer.

[0169] Event 2 can be used to launch a game application. For example, Event 2 can be a user's click event on the icon of a video application, as described in Figure 1.

[0170] A game layer creation request can be used to request the creation of a game layer for displaying any interface in a game application. The game layer creation request can include: the identifier of the game application, such as the package name of the game application.

[0171] S604, WMS, and SurfaceFlinger simultaneously create video window layers and video layers, and add layer properties to each layer.

[0172] Layer properties can include one or more of the following: the rounded corners of the layer, the position of the layer, or the size of the layer.

[0173] The layer properties of the video window layer can include one or more of the following: rounded corners, position of the video window layer, and size of the video window layer.

[0174] The layer properties of a game layer can include one or more of the following: the position of the video layer, and the size of the video window layer, etc.

[0175] The game layer's layer properties can also include the position information of the video window. Adding this position information to the game layer's properties allows the electronic device to process the game layer based on the video window's location.

[0176] It should be noted that WMS can simultaneously create each layer and add layer properties within SurfaceFlinger.

[0177] Through the steps S602-S604 described above, the electronic device can create the various layers included in an image frame.

[0178] It should be noted that after S602, in response to the request to create a video window layer, WMS and SurfaceFlinger can execute the steps in S604 of creating a video window layer and adding layer attributes to the video window layer; after S603, in response to the request to create a game layer, WMS and SurfaceFlinger can execute the steps in S604 of creating a game layer and adding layer attributes to the game layer. That is, this embodiment does not limit the order of execution between S602 and S604.

[0179] S605 and SurfaceFlinger detect the presence of rounded corner windows based on the layer properties of the video window layer.

[0180] For example, during the composition information detection phase of each frame, SurfaceFlinger monitors whether the layers involved in the composition include rounded corner layers. If SurfaceFlinger can detect that the video window layer has rounded corner attributes, it determines that a rounded corner window exists and then executes S606. This application embodiment does not limit the method of detecting the existence of a rounded corner window.

[0181] Optionally, if SurfaceFlinger does not detect a rounded corner window, SurfaceFlinger can call HWC to overlay the received layers to obtain an image frame, which is then sent to the display screen via other modules for display. In this scenario, the electronic device does not need to perform the steps shown in S606-S620, and the electronic device does not need to call the GPU to perform the compositing of the rounded corner effect.

[0182] S606 and SurfaceFlinger cache the position information of the video window and generate message 1.

[0183] Message 1 can be used to request the GUE service to detect whether the preset switch is enabled. The preset switch can be used to start the image processing process in the first scene.

[0184] During image processing in the first scenario, the electronic device can achieve rounded corner effects by detecting that the first scenario is met, through methods such as not rendering pixels in the area where the video window is located in the game layer, or making the area where the video window is located transparent.

[0185] S607, SurfaceFlinger sends message 1 to the GUE service, containing the location information of the video window and the identifier of the game application.

[0186] S608 and GUE services respond to message 1 and determine whether to enable the preset switch based on the system status and / or game status.

[0187] System status can be used to indicate the operating state of an electronic device. System status may include one or more of the following: power saving state, device rotation state, rounded window movement state, or multi-game state, etc.

[0188] Among them, the power saving state can be the state entered when the battery level of the electronic device is less than 10%, or it can be the state entered after the user turns on the low power mode. The power saving state can extend the standby time of the device; the device rotation state can include: the state of the device switching from landscape to portrait mode, and / or the state of the device switching from portrait to landscape mode; the motion state of the rounded corner window can include: the state when the rounded corner window remains unchanged for less than 1 hour, such as the state of the rounded corner window scaling (i.e., the size of the rounded corner window changes), and / or the state of the rounded corner window moving, etc.; the multi-game state can be understood as the electronic device detecting at least two application windows that are different game application windows.

[0189] For example, let's take the case where the GUE service detects a system state including a device switching from portrait to landscape mode. The electronic device executes S609-S612 and completes the cutout based on S611 or S612 in portrait mode. After switching from portrait to landscape mode, the cutout position remains the same, but the position of the video window changes accordingly. At this time, the position where the video window should be displayed differs from the position of the cutout, which will cause the displayed content to be disordered.

[0190] Game status can be used to indicate the running state of a game application. For example, game status may include frame interpolation status and / or non-fullscreen status.

[0191] Frame interpolation state can be understood as the state when frame interpolation is triggered in a game application, such as triggering a frame rate change interpolation button in the game assistant of a game application; non-fullscreen state can be understood as detecting that the window corresponding to the game application is not in fullscreen display.

[0192] For example, let's take the case where the GUE service detects a non-full-screen state (such as a game application being located in the upper half of the screen) in the system state. The electronic device executes S609-S612, and performs a hole-punch operation based on S611 or S612 when the game application is in full-screen state. After switching from full-screen state to non-full-screen state, the hole-punch position remains the same, but the position of the video window changes. At this time, there is a difference between the position where the video window should be displayed and the hole-punch position. For example, the video window may be located in the upper half of the screen, and the hole-punch position may be located in the middle of the screen, resulting in disordered display content.

[0193] It is understood that the system status can be read by the GUE service from modules such as the system server through interface calls; the game status can be read by the GUE service from the game application or other modules monitoring the game application through interfaces. This application embodiment will not elaborate on the acquisition methods of the system status and game status.

[0194] Determining the conditions for enabling the preset switch based on the system state and / or game state may include: not detecting all states in the system state, and / or not detecting all states in the game state. When the GUE service determines to enable the preset switch based on the system state and / or game state, the GUE service may send a message to surfaceflinger instructing the preset switch to be enabled and perform the steps shown in S609-S620.

[0195] The conditions for determining whether to enable the preset switch based on the system state can include one or more of the following: detecting any of the system states and / or detecting any of the game states. If the GUE service determines that the preset switch should not be enabled based on the system state and / or game state, the GUE service can send message 2 to surfaceflinger. Message 2 can be used to indicate that the preset switch is not enabled. In response to message 2, surfaceflinger can invoke the GPU to perform the compositing of the rounded corner effect in the video window.

[0196] It is understood that electronic devices can improve the accuracy of image processing methods by detecting the game state and / or system state in S608, that is, to perform image processing in the first scene in the most suitable state of the electronic device. Furthermore, the game state and system state provided in this application embodiment are merely examples and do not constitute a limitation on the embodiments of this application.

[0197] S609, GUE service sends the location information of the video window and the identifier of the game application to gamesware.

[0198] S610 and Gamesaware detect whether the game application's identifier is in whitelist 2.

[0199] Whitelist 2 can be used to store the package names of applications that are allowed to execute Scheme 1.

[0200] If gamesaware detects that the identifier of the game application is in whitelist 2, gamesaware may perform the step shown in S611; or, if gamesaware detects that the identifier of the game application is not in whitelist 2, gamesaware may perform the step shown in S612.

[0201] Optionally, the GUE service can also send the position information of the video window to gamesaware in S609, so that gamesaware can perform the image processing steps in S611 or S612 without performing the judgment in S610.

[0202] S611 and gamesaware perform pixel rendering on areas other than the video window in the game layer to obtain the processed game layer (or Scheme 1).

[0203] As can be understood, as shown in Figure 7, before executing S611, gamessaware can obtain the game layer 701 from surfaceflinger. During the execution of S611, gamessaware can perform pixel rendering on the areas other than the video window in game layer 701 based on the position information of the video window, to obtain the processed game layer (processed game layer 702 in Figure 7).

[0204] The processed game layer 702 may include a rounded rectangle area 703, the location of which is the location of the game window. The rounded rectangle area 703 may be white or black to indicate that it does not contain any pixels.

[0205] Based on Option 1, it is understandable that gamesaware does not perform pixel rendering of the video window, which allows electronic devices to save power during the rendering process of game screens.

[0206] S612 and gamesaware perform pixel rendering on the game layer and make the video window in the game layer transparent to obtain the processed game layer (or Scheme 2).

[0207] Transparency can be understood as setting the transparency of an area to 100%, such as setting the transparency of the video window in a game layer to 100%.

[0208] As can be understood, as shown in Figure 7, before executing S611, gamessaware can obtain the game layer 701 from surfaceflinger. During the execution of S612, after gamessaware renders all pixels in the game layer, it can set the transparency of the video window's location in game layer 701 to 100% based on the video window's position information, thus obtaining the processed game layer (processed game layer 702 in Figure 7). At this time, the transparency of other areas in the game layer, except for the video window, can be 0.

[0209] The processed game layer 702 may include a rounded rectangular area 703, the location of which is the location of the game window. If image 1 is contained within the rounded rectangular area 703 of the game layer, the electronic device can make it transparent so that image 1 is not visible within the rounded rectangular area 703. In this case, the rounded rectangular area 703 can appear white or black to indicate that it has been made transparent. Image 1 is not shown in Figure 7.

[0210] Based on Solution 2, gamesaware makes the video window in the game layer transparent, allowing electronic devices to avoid compositing rounded corners in the GPU, thus saving power consumption.

[0211] Optionally, after performing the pixel rendering process in S611 or S612, gamessaware can also continue to perform image processing steps such as Gaussian blurring and color processing on the rendered game layer to obtain a game layer with better display effect.

[0212] It should be noted that before implementing Option 1, the pixel rendering method for game layers in Gamesaware needs to be modified to achieve the rounded corner effect; before implementing Option 2, the pixel rendering method does not need to be modified, but a transparency process is added after the pixel rendering and other layer processing is completed to achieve the rounded corner effect.

[0213] S613, gamesaware sends message 3 and the processed game layer to surfaceflinger via GUE service.

[0214] Message 3 can be used to indicate the completion of the rounded corner effect composition.

[0215] S614, surfaceflinger responds to message 3 and cancels rounded corner composition.

[0216] It is understandable that the layers in the target image frame 1 may include: the processed game layer and the video window layer. After receiving message 3, SurfaceFlinger can determine that the rounded corner effect has been completed. Then, SurfaceFlinger can perform the steps of canceling the rounded corner composition and refreshing the video window layer in S614-S615, and present the rounded corner effect by overlaying the processed game layer and the video window layer in S616-S618.

[0217] Optionally, when the target image frame 1 includes layers other than the processed game layer, SurfaceFlinger can receive message 3 corresponding to all other layers, indicating that the rounded corner effect can be completed for all other layers. Therefore, SurfaceFlinger can cancel the rounded corner compositing, that is, SurfaceFlinger will not call the GPU to perform the rounded corner effect compositing, so as to reduce the GPU load.

[0218] S615, SurfaceFlinger refreshes the video content of the video window layer.

[0219] SurfaceFlinger can refresh the content of the video window layer, i.e., the video content, so that the video window layer can display the latest video frames.

[0220] For example, after decoding the latest video frame, the codec component can send the video frame to SurfaceFlinger, which can then refresh the video window layer. It should be noted that the timing of SurfaceFlinger refreshing the video window layer is not limited to that shown in Figure 6. For instance, the codec component can decode a video frame according to the needs of the video player and send it to SurfaceFlinger, which can then refresh the video window layer upon receiving the frame. Of course, in multiple consecutive image frames, the video content displayed in the video window may be static. For example, if playback is paused, the video content remains unchanged and is static. In this case, SurfaceFlinger does not need to refresh the video window layer for every frame. That is, refreshing the video window layer is not necessary for every frame.

[0221] S616 and SurfaceFlinger retrieve the stacking order of the processed game layer and video window layer, with the video window layer at the top.

[0222] S617 and SurfaceFlinger adjust the video window layer from the top layer to the bottom layer to obtain the adjusted stacking order.

[0223] In this way, SurfaceFlinger can overlay the video window layer on the bottom layer, thus achieving a rounded corner effect by using the outline of the rounded rectangle area in the processed game layer.

[0224] S618 and SurfaceFlinger combine the processed game layer and video window layer according to the adjusted overlay order to obtain target image frame 1.

[0225] Because rounded corner compositing is disabled, SurfaceFlinger does not call the GPU to crop the video window layer to achieve the rounded corner effect, thus reducing the GPU load. For example, SurfaceFlinger can call HWC to complete the layer overlay and obtain the target image frame 1.

[0226] As shown in Figure 7, after compositing the game layer 702 and the video window layer 704 in the order of stacking, a target image frame 705 can be obtained. The rounded rectangular area of ​​the target image frame 705 can display part of the content in the video window layer 704.

[0227] In S606-S618 above, the case where the game layer has completed the hole-cutting was explained. Optionally, it is also possible that other layers besides the game layer may not have completed the hole-cutting. In this case, SurfaceFlinger can determine that message 3 has not been received, and therefore does not cancel the rounded corner compositing. Subsequently, SurfaceFlinger can composite the processed game layer and video window layer according to the unadjusted stacking order to obtain target image frame 1, where SurfaceFlinger calls the GPU to complete the rounded corner compositing, so that the video window in target image frame 1 has a rounded corner effect.

[0228] S619, SurfaceFlinger sends target image frame 1 to the display.

[0229] It is understandable that SurfaceFlinger does not send the target image frame 1 directly to the display screen, but rather transmits and processes it through multiple software modules so that the display screen receives the target image frame 1.

[0230] S620, The display shows target image frame 1.

[0231] For example, the display shows the next image frame after each refresh.

[0232] As can be understood, the embodiment corresponding to Figure 6 illustrates the processing of an image frame after receiving event 1 and event 2. In practice, before closing the rounded corner window, each image frame can be processed in a similar manner, thereby achieving a rounded corner effect in each image frame. That is, the phone can listen for the presence of a rounded corner window while processing each image frame. If a rounded corner window is detected, the rounded corner effect will be composited when drawing other layers.

[0233] In one specific implementation, in the process shown in Figure 6, S605-S620 can be executed cyclically, thereby reducing the GPU load in each image frame.

[0234] Based on the description in Figure 6, electronic devices can achieve rounded corner effects by either not rendering the area containing the rounded corner window in the game layer as pixels, or by making the area containing the rounded corner window transparent, thus replacing GPU-composited rounded corner effects. In this way, the rounded corner effect does not need to be achieved through GPU compositing, reducing the GPU load during image processing.

[0235] It should be noted that when electronic devices monitor the image processing process through commands such as the `gump` command to the `surfaceflinger` or tracing technology, the electronic devices can detect that the process of obtaining the target image frame 1 based on the video window layer and the game layer passes through HWC and does not pass through the GPU.

[0236] Both `gump surfaceflinger` and `trace` can be understood as tools used to analyze the performance of image processing. The `gump surfaceflinger` command can be used to analyze information such as frame rate changes, rendering time, and compositing details related to SurfaceFlinger; `trace` can be used to analyze the resource usage, thread scheduling, system performance, and state changes of electronic devices. Electronic devices can obtain detailed information on CPU, GPU, and memory usage, as well as program execution details through `trace`.

[0237] Based on the description in Figure 6, when the rounded corner window is activated, the electronic device can also synthesize the rounded corner effect by calling the GPU when event 3 is detected, as described in Figures 8-9.

[0238] Event 3 may include one or more of the following: an event triggered by the motion state of the rounded corner window, or an event triggered by the device rotation state. Among them, the event triggered by the motion state of the rounded corner window may include: an event generated by the user's drag operation on the rounded corner window, and an event generated by the user's scaling operation on the rounded corner window (such as a double-click operation, or a drag operation on the edge).

[0239] In the embodiments corresponding to Figures 8 and 9, Event 3 is an example of an event generated by the user's drag operation on the rounded corner window (or a drag operation event). This example does not constitute a limitation on the embodiments of this application.

[0240] After launching the video window, the electronic device can display the interface shown in Figure 8A, which includes the video window 8011. In response to the user dragging the video window 8011 along the direction of the arrow in Figure 8A, the electronic device can adjust the position of the video window 8011. After adjustment, the electronic device can display the interface shown in Figure 8B. Figure 8B also includes the video window 8011. Unlike Figure 8A, the position of the video window 8011 in Figure 8B has changed.

[0241] In some embodiments, for the above-mentioned adjustment of rounded corner windows, during the adjustment process, such as adjusting from the position of video window 8011 in Figure 8A to the position of video window 8011 in Figure 8B, the rounded corner effect is achieved by the GPU through rounded corner compositing. After the adjustment is completed and stabilized, such as after adjusting to the position of video window 8011 in Figure 8B, the rounded corner effect can be achieved through Scheme 1 or Scheme 2 described in Figure 6.

[0242] It should be noted that during the adjustment of rounded corner windows, if the rounded corner effect is achieved using either Solution 1 or Solution 2, there may be a delay. For example, when Event 3 is received during the display of the k-th frame, SurfaceFlinger, after receiving the updated position information of the video window, can send it to GameSaware via the GUE service. However, before receiving the updated position information of the video window, GameSaware may already be performing pixel rendering of the game layer in the (k+1)-th frame. In other words, GameSaware may execute Solution 1 or Solution 2 based on the unupdated position information of the video window, resulting in a delay in the position information of the video window used for "hole-cutting," or simply, a "hole-cutting delay."

[0243] Subsequently, when SurfaceFlinger performs layer compositing, there may be inconsistencies between the rounded rectangle area determined by the cutout and the area where the rounded layer is located, such as different positions. This can lead to issues such as black areas in the rounded rectangle area and incomplete display of content in the video window.

[0244] Taking the example of dragging the video window 8011 from the position of the video window 8011 in the interface 801 shown in Figure 8A in the direction indicated by the arrow, if the first or second method is used during the dragging process, the interface shown in Figure 8C may be displayed. In Figure 8C, part of the video window 8011 is black, and the content in the video window 8011 only shows the part of the person, which is missing.

[0245] Based on the problems that arise during the adjustment of rounded corner windows, this embodiment adopts the method of using the GPU to perform rounded corner compositing to achieve the rounded corner effect during the adjustment of rounded corner windows, which can avoid the above-mentioned problems caused by the delay in punching holes.

[0246] The image processing procedure when the electronic device detects event 3 will be illustrated below with reference to the embodiment corresponding to Figure 9.

[0247] Figure 9 is a schematic diagram of the module interaction of another image processing method provided in an embodiment of this application. As shown in Figure 9, after S604, the image processing method may further include the following steps:

[0248] S901 and SurfaceFlinger detect the presence of rounded corner windows based on the layer properties of the video window layer.

[0249] After S901, electronic devices can determine whether to call the GPU to composite rounded corner effects through scheme A and / or scheme B.

[0250] In Scheme A, SurfaceFlinger can determine whether to call the GPU to composite the rounded corner effect by detecting the duration for which the video window remains unchanged, as described in S902, S606-S620, and S903-S905.

[0251] S902, SurfaceFlinger detects whether the continuous duration of a video window layer remaining unchanged exceeds duration 1.

[0252] For example, if the continuous duration during which the video window layer remains unchanged as detected by SurfaceFlinger exceeds duration 1, the electronic device may perform the steps shown in S606-S620; or, if the continuous duration during which the video window layer remains unchanged as detected by SurfaceFlinger does not exceed duration 1, the electronic device may perform the steps shown in S903.

[0253] S903, SurfaceFlinger sends message 4 to GUE service.

[0254] Message 4 can be used to indicate whether to cancel the rounded corner effect in a composition.

[0255] S904, GUE service sends message 4 to gamesware.

[0256] S905 and gamesaware respond to message 4 by performing pixel rendering on the game layer to obtain the rendered game layer, and either not executing option one or option two.

[0257] After receiving message 4, gamesaware can determine to cancel the rounded corner effect composition, thus eliminating the need for pixel rendering of other areas in the game layer except for the game window (i.e., not executing option one), or eliminating the need to make the area where the game window is located in the game layer transparent (i.e., not executing option two).

[0258] In Solution B, SurfaceFlinger can detect whether the position of the video window has moved through the GUE service, and then determine whether to call the GPU to composite the rounded corner layer, as described in S606-S608 and S906-S908.

[0259] Prior to S906, the GUE service could determine whether to turn on the preset switch based on the motion state of the rounded window in S908. Therefore, the GUE service could execute the steps shown in S906.

[0260] S906, GUE service sends message 2 to surfaceflinger.

[0261] The meaning of message 2 can be found in the description in S608, and will not be repeated here.

[0262] S907, GUE service sends message 5 to gamesaware.

[0263] Message 5 can be used to indicate whether to cancel the rounded corner effect in a composition.

[0264] S908 and gamesaware respond to message 5 by performing pixel rendering on the game layer to obtain the rendered game layer, and by not executing either option one or option two.

[0265] The steps in S908 can be found in the description in S905, and will not be repeated here.

[0266] S909, gamesaware sends message 6 and the rendered game layer to surfaceflinger via GUE service.

[0267] Message 6 can be used to indicate that a composition with an incomplete rounded corner effect is not yet complete.

[0268] S910 and SurfaceFlinger respond to message 6 without canceling rounded corner composition.

[0269] Understandably, SurfaceFlinger can determine from message 6 that the rounded corner effect has not yet been composited, and therefore can determine that the rounded corner effect will be composited by calling the GPU in the future.

[0270] S911, SurfaceFlinger refreshes the video content of the video window layer.

[0271] S912 and SurfaceFlinger obtain the overlay order of the rendered game layer and video window layer, with the video window layer at the top.

[0272] It is understandable that the rendered game layer is different from the processed game layer described in Figure 6. For example, the rendered game layer does not process the position of the video window, which is equivalent to not performing the rounded corner effect compositing; while the processed game layer, after going through Scheme 1 or Scheme 2, can realize the creation of holes in the game layer, which is equivalent to completing the rounded corner effect compositing.

[0273] S913 and SurfaceFlinger call the GPU to synthesize rounded corner effects, and synthesize the processed game layer and video window layer according to the stacking order to obtain target image frame 2.

[0274] Surfaceflinger uses the GPU to round the corners of the video window layer, and then overlays the video window layer on top of the disc player in the order of the discs to ensure that the video window is visible.

[0275] S914, SurfaceFlinger sends target image frame 2 to the display screen.

[0276] S915, The display screen shows target image frame 2.

[0277] For example, the display shows the next image frame after each refresh.

[0278] Based on the description in Figure 9, when the electronic device can detect the existence of a rounded corner window and the position of the rounded corner window changes, the electronic device does not need to execute Scheme 1 or Scheme 2 through gamesaware. Instead, it uses the GPU to synthesize the rounded corner effect and overlays the video window layer with the rounded corner effect on the top layer to obtain the target image frame 2.

[0279] Based on the description in Figure 6, when the rounded corner window is activated, the electronic device can also cancel the composition of the rounded corner effect when event 4 is detected, as described in Figures 10-11.

[0280] Event 4 may include events such as closing a rounded window, such as events triggered by the user clicking the close button in the rounded window, or events triggered by the user dragging the rounded window to the edge of the display screen.

[0281] In the embodiments corresponding to Figures 10 and 11, event 4 is an example of an event triggered by the user clicking the close button in the rounded corner window. This example does not constitute a limitation on the embodiments of this application.

[0282] After the video window is launched, in response to the user's click operation on the video window, the electronic device can display the interface shown in Figure 10A. Figure 10A may include the video window 1001, and the video window may include a close button 1002.

[0283] In response to a user clicking the close button 1002, the electronic device can close the video window 1001 and display the interface shown in Figure 10B. The video window 1001 is no longer included in Figure 10B.

[0284] The image processing procedure when the electronic device detects event 4 will be illustrated below with reference to the embodiment corresponding to Figure 11.

[0285] Figure 11 is a schematic diagram of the module interaction of another image processing method provided in an embodiment of this application. As shown in Figure 11, the image processing method may further include the following steps:

[0286] S1101, In response to event 4, the video player sends a request to the WMS to destroy the video window layer.

[0287] The video pop-up layer destruction request can be used to request the destruction of the video pop-up layer.

[0288] S1102, WMS and SurfaceFlinger synchronously destroy the video window layer.

[0289] For example, after WMS receives a request to destroy the video window layer, WMS can destroy the video window layer and simultaneously notify SurfaceFlinger to destroy the video window layer. In this way, SurfaceFlinger can listen for the result of S1103 as follows.

[0290] S1103, SurfaceFlinger cannot detect the presence of rounded corner windows.

[0291] The method for SurfaceFlinger to monitor the presence of rounded corner windows can be found in the description in S605, and will not be repeated here.

[0292] Understandably, SurfaceFlinger has been listening continuously since power-on, detecting the presence or absence of rounded corner windows for each image frame. Specifically, after receiving event 4, the electronic device closes the video window, and SurfaceFlinger will detect the absence of a rounded corner window.

[0293] SurfaceFlinger can detect the absence of rounded corner windows after the video window layer is destroyed and before the video window (or other rounded corner windows) is restarted.

[0294] S1104, SurfaceFlinger sends message 7 to Gamesware via GUE service.

[0295] Message 7 can be used to indicate whether to cancel the rounded corner effect in a composition.

[0296] S1105, in response to message 7, perform pixel rendering on the game layer to obtain the rendered game layer, and do not execute either Option 1 or Option 2.

[0297] S1106, GameSaware sends message 8 and the rendered game layer to SurfaceFlinger via the GUE service.

[0298] Message 8 can be used to indicate that the rounded corner effect is not yet complete in the composition, as well as the completion of the drawing of the game layer.

[0299] S1107, surfaceflinger responds to message 8, obtains the rendered game layer, and obtains the target image frame 3 based on the rendered game layer.

[0300] Understandably, since the video window is closed at this point, the electronic device does not need to perform rounded corner compositing through either Solution 1 or Solution 2, nor does it need to call the GPU to compose the rounded corner effect. At this time, SurfaceFlinger can obtain the target image frame 3 based on the rendered game layer, or it can perform other processing such as merging the rendered game layer with other layers to obtain the target image frame 3.

[0301] S1108, SurfaceFlinger sends target image frame 3 to the display.

[0302] S1110, The display screen shows the target image frame 3.

[0303] The embodiments in Figures 10 and 11 above illustrate the processing of an image frame after receiving event 4. In practice, after closing the rounded corner window and before reopening it, the processing of each image frame can be done in a similar manner, so that the rounded corner effect is not composited and the stacking order between layers is not changed during the processing of each image frame.

[0304] For example, in the embodiment corresponding to Figure 11, the electronic device can execute S1103-S1109 in a loop, so that in each image frame, if the absence of a rounded corner window is detected, the rounded corner effect is not synthesized, and the overlay order between layers is not changed.

[0305] Referring to the embodiment corresponding to Figure 12, the electronic device can determine whether it is currently in the first scene or the second scene by detecting the preset layer (i.e., S605), and then execute the image processing process in the first scene or the image processing process in the second scene.

[0306] The image processing procedure for the first scenario can be found in the descriptions in S606-S620; the image processing procedure for the second scenario can be found in the descriptions in S1205-S1216.

[0307] Figure 12 is a schematic diagram of the module interaction of another image processing method provided in an embodiment of this application. As shown in Figure 12, the image processing method may include the following steps:

[0308] Following S601 and S602, in S1201, in response to event 5, the target application sends a request to the WMS to create the target layer.

[0309] Event 5 can be used to launch the target application, such as when the user clicks on the icon of the target application.

[0310] The target application can be a game application as described in Figures 6-11, or it can be a desktop application, etc.

[0311] The target layer creation request can be used to request the creation of a target layer for displaying any interface in the target application.

[0312] S1202, WMS, and SurfaceFlinger simultaneously create video window layers and target layers, and add layer properties to each layer.

[0313] The layer properties of the target layer can include attributes such as the position of the target layer and the size of the target layer.

[0314] S1203 and SurfaceFlinger detect the presence of rounded corner windows based on the layer properties of the video window layer.

[0315] The process of SurfaceFlinger listening for the presence of rounded corner windows can be found in the description in S605, and will not be repeated here.

[0316] S1204, surfaceflinger determines whether a preset layer has been detected.

[0317] The preset layers may include layers that allow rounded corners to be achieved through either Option 1 or Option 2. For example, the preset layers may include game layers.

[0318] For example, when the target application is a game application, SurfaceFlinger can determine that the game layer corresponding to the game application has been detected; or, when the target application is a desktop application, SurfaceFlinger can determine that no preset layer has been detected.

[0319] If the surfaceflinger detects a preset layer, the electronic device can perform the steps shown in S606-S620, that is, perform the image processing process in the first scene; if the surfaceflinger does not detect a preset layer, the electronic device can perform the steps shown in S1205-S1216, that is, perform the image processing process in the second scene.

[0320] S1205, SurfaceFlinger generates message 9, which caches the position information of the video window.

[0321] Message 9 can be used to indicate how to achieve a rounded corner effect by drawing a transparent rounded rectangle area.

[0322] S1206, SurfaceFlinger sends message 9 to libhwui, along with the position information of the video window.

[0323] S1207. The target application sends the content to be drawn in the target application to libhwui.

[0324] It is understood that the target application can actively send the content to be drawn 1 to libhwui, or libhwui can obtain the content to be drawn 1 from the target application through an interface call. This application embodiment does not limit this.

[0325] S1208, in response to message 9, libhwui draws the content to be drawn 1 in the target layer and draws a rounded rectangle area based on the position information of the video window, thus obtaining the processed target layer.

[0326] For example, when the target application is a desktop application, content 1 can be desktop icons and / or desktop wallpapers, etc.

[0327] Normally, even if a rounded corner window exists in an image frame, the drawing phase mainly focuses on drawing the layer content, without needing to implement the rounded corner effect. However, in this embodiment, after receiving the content to be drawn (1), libhwui can draw content 1 in the target layer. Furthermore, libhwui also receives message 9. Therefore, after drawing content 1 in the target layer, libhwui can determine the drawing of a transparent rounded rectangle area based on the position information of the video window, and calculate the position, size, and corner radius of the rounded rectangle area to complete the drawing.

[0328] Optionally, libhwui can also calculate the position, size, and corner radius of the rounded rectangle region based on more information. This information can be passed from surfaceflinger to libhwui via the transmission path shown in S1206. This information may include: device status information, the scaling ratio of the desktop layer, etc. Alternatively, this information may be obtained by libhwui from other modules, such as obtaining the length and width information of the display screen from memory. This application does not specifically limit this.

[0329] S1209, libhwui sends message 10 to surfaceflinger, and the processed target layer.

[0330] Message 10 is used to indicate the completion of the rounded corner effect composition.

[0331] S1210, SurfaceFlinger responds to message 10 and cancels rounded corner composition.

[0332] It is understandable that after receiving message 10 from libhwui, it indicates that the electronic device has achieved a rounded corner effect through a transparent rounded rectangle area, so surfaceflinger cancels the rounded corner composition.

[0333] S1211, SurfaceFlinger refreshes the video content of the video window layer.

[0334] S1212, surfaceflinger obtains the stacking order of the processed target layer and the video window layer, where the video window layer is located on top.

[0335] S1213, surfaceflinger adjusts the video window layer from the top layer to the bottom layer to obtain the adjusted stacking order.

[0336] S1214, surfaceflinger combines the processed target layer and video window layer according to the adjusted overlay order to obtain target image frame 4.

[0337] S1215, SurfaceFlinger sends target image frame 4 to the display.

[0338] S1216, The monitor displays target image frame 4.

[0339] The steps shown in S1211-S1216 are similar to those described in S615-S620, and will not be repeated here.

[0340] Based on the descriptions in S1205-S1216, since the rounded corner effect has already been composited in the libhwui corresponding to the target application, surfaceflinger cancels the rounded corner compositing. Surfaceflinger will not call the GPU to crop the video window layer to achieve the rounded corner effect, thereby reducing the GPU load.

[0341] Based on the description in Figure 12, the electronic device can identify the scene it is in by judging the preset layer, and determine whether to execute the image processing process in the first scene or the second scene, thereby enriching the application scenarios of the image processing method and improving the stability and accuracy of the image processing method.

[0342] It should be noted that the sequential relationship between the steps in Figures 6, 9, 11 and 12 in the embodiments of this application is only an example and does not constitute a limitation on the embodiments of this application.

[0343] It should be noted that the interface provided in this application embodiment is only an example and does not constitute a limitation on the embodiments of this application.

[0344] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0345] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0346] Based on the descriptions in Figures 6-12, the image processing method will be illustrated below with reference to the embodiment corresponding to Figure 13. Figure 13 is a schematic flowchart of an image processing method provided by an embodiment of this application.

[0347] As shown in Figure 13, the image processing method may include the following steps:

[0348] S1301, run the first and second applications.

[0349] The first application corresponds to the first window, which is a non-full-screen window. The second application corresponds to the second window, which is a full-screen window. The second application is a game application.

[0350] The first application is a video application (or video player), the second application is a game application, and the first layer is a video window layer.

[0351] S1302, Get the first layer, get the second layer.

[0352] Specifically, obtaining the second layer includes: performing pixel rendering on the second region of the second window to obtain the second layer, or performing pixel rendering on the first region and the second region of the second window to obtain the pixel-rendered second window, and using the position of the first window to make the first region of the pixel-rendered second window transparent to obtain the second layer.

[0353] The second layer is the processed game layer, while the first layer can be the video window layer.

[0354] S1303. Combine the first layer corresponding to the first window and the second layer corresponding to the second window to obtain the first image.

[0355] The second window includes a first region and a second region. The first region is the rounded corner region in the second window that corresponds to the position of the first window, and the second region is the region in the second window other than the first region.

[0356] As shown in Figure 7, the first region can be region 703, and the second region can be any region other than region 703.

[0357] S1304, Display the first image.

[0358] The first image can be target image frame 1, such as target image frame 705 in Figure 7.

[0359] In S1301-S1304, the electronic device can perform image processing in the first scene based on the embodiment corresponding to FIG6.

[0360] S1305, In response to the operation of exiting the second window, run the first application and the third application.

[0361] The third application corresponds to the third window, which is a full-screen window, and the third application is a desktop application.

[0362] Exiting the second window closes the game application; the third application can be a desktop application.

[0363] S1306, Get the fourth layer, get the third layer.

[0364] S1307. Combine the fourth layer corresponding to the first window and the third layer corresponding to the third window to obtain the second image.

[0365] The third window includes a third region and a fourth region. The third region is the rounded corner area in the third window that corresponds to the position of the first window, and the fourth region is the area in the third window other than the third region.

[0366] S1308, Display the second image.

[0367] The second image can be the image shown in Figure 1B.

[0368] In S1305-S1308, the electronic device can perform image processing in the second scenario based on the steps described in S1205-S1216.

[0369] Based on this, electronic devices can avoid using the GPU to composite the rounded corners of the first window. Instead, they can reserve a rounded corner area in the second window and then composite the layers corresponding to the two windows respectively. This allows the content in the first window to be displayed in the rounded corner area after the layers are composited, thus saving GPU power consumption.

[0370] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above list sorting method.

[0371] Figure 14 is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application.

[0372] The electronic device includes a processor 1401, a communication line 1404, and at least one communication interface (in FIG14, communication interface 1403 is used as an example for illustration).

[0373] Processor 1401 may be a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs according to the present application.

[0374] Communication line 1404 may include circuitry for transmitting information between the aforementioned components.

[0375] Communication interface 1403 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0376] Possibly, the electronic device may also include a memory 1402.

[0377] The memory 1402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 1404. The memory may also be integrated with the processor.

[0378] The memory 1402 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 1401. The processor 1401 executes the computer execution instructions stored in the memory 1402 to implement the method provided in the embodiments of this application.

[0379] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0380] In a specific implementation, as one embodiment, processor 1401 may include one or more CPUs, such as CPU0 and CPU1 in FIG14.

[0381] In a specific implementation, as one embodiment, the electronic device may include multiple processors, such as processor 1401 and processor 1405 in FIG. 14. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0382] The image processing method provided in this application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0383] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0384] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0385] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0386] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0387] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0388] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0389] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: Run a first application and a second application. The first application corresponds to a first window, which is a non-full-screen window. The second application corresponds to a second window, which is a full-screen window. The second application is a game application. The first layer corresponding to the first window and the second layer corresponding to the second window are composited to obtain a first image. The second window includes a first region and a second region. The first region is the rounded corner region in the second window corresponding to the position of the first window, and the second region is the region in the second window other than the first region. Display the first image.

2. The method according to claim 1, characterized in that, The method further includes: Get the first layer, and then get the second layer.

3. The method according to claim 2, characterized in that, The step of obtaining the second layer includes: Pixel rendering is performed on the second area of ​​the second window to obtain the second layer.

4. The method according to claim 2 or 3, characterized in that, The step of obtaining the second layer includes: Pixel rendering is performed on the first region and the second region of the second window to obtain the pixel-rendered second window, and the first region of the pixel-rendered second window is made transparent using the position of the first window to obtain the second layer.

5. The method according to any one of claims 2-4, characterized in that, The step of obtaining the second layer includes: Obtain the system status of the electronic device; If the system is detected to be not in a power-saving state, the second layer is obtained.

6. The method according to any one of claims 2-5, characterized in that, The step of obtaining the second layer includes: Obtain the system status of the electronic device; If the system is not in a device rotation state, the second layer is obtained. The device rotation state includes: the device switching from landscape to portrait mode, and / or the device switching from portrait to landscape mode.

7. The method according to any one of claims 2-6, characterized in that, The step of obtaining the second layer includes: Obtain the motion state of the first window; If the first window is not in motion, the second layer is acquired. The first window being in motion includes: the size of the first window changing, and / or the position of the first window changing.

8. The method according to any one of claims 2-7, characterized in that, The electronic device includes: an image synthesizer, application awareness, and a hardware synthesizer (HWC). After running the first application and the second application, the method further includes: when the image synthesizer detects the presence of a rounded corner window, the image synthesizer sends the position of the first window and the identifier of the second application to the application perception. The step of obtaining the second layer includes: the application awareness obtaining the second layer based on the position of the first window; The method further includes: the application awareness sending the second layer and the first message to the image synthesizer; In response to the first message, the image synthesizer does not invoke the image processor GPU to synthesize the rounded corner effect of the first window; The step of compositing the first layer corresponding to the first window and the second layer corresponding to the second window includes: the image compositer calling the HWC to composite the first layer and the second layer.

9. The method according to claim 8, characterized in that, The electronic device also includes: GUE services. The image synthesizer sends the position of the first window and the identifier of the second application to the application perception, including: The image synthesizer sends the position of the first window and the identifier of the second application to the GUE service; When the GUE service detects that the system status of the electronic device and / or the running status of the second application meet preset conditions, the GUE service sends the position of the first window and the identifier of the second application to the application perception.

10. The method according to claim 8 or 9, characterized in that, The image synthesizer sends the position of the first window and the identifier of the second application to the application perception, including: When the image synthesizer detects a preset layer, the image synthesizer sends the position of the first window and the identifier of the second application to the application perception, wherein the preset layer includes the layer corresponding to the game application.

11. The method according to any one of claims 1-10, characterized in that, The process of compositing the first layer and the second layer passes through the HWC of the electronic device.

12. The method according to any one of claims 1-11, characterized in that, During the process of compositing the first layer and the second layer, the second layer is superimposed on top of the first layer.

13. The method according to any one of claims 1-12, characterized in that, After obtaining the first image, the method further includes: In response to the operation of exiting the second window, the first application and the third application are run, the third application corresponds to the third window, the third window is a full-screen window, and the third application is a desktop application; The fourth layer corresponding to the first window and the third layer corresponding to the third window are composited to obtain a second image. The third window includes a third region and a fourth region. The third region is the rounded corner region in the third window that corresponds to the position of the first window. The fourth region is the region in the third window other than the third region. The second image is displayed.

14. The method according to claim 13, characterized in that, The method further includes: obtaining the fourth layer and obtaining the third layer; The process of obtaining the third layer includes: drawing a transparent rounded rectangle in the third region and drawing the first content to be displayed in the fourth region, thereby obtaining the third layer.

15. The method according to claim 13, characterized in that, The electronic device includes: an image synthesizer, a rendering module, and an HWC; After running the first application and the third application, the method further includes: when the image synthesizer detects the presence of a rounded corner window, the image synthesizer sends a second message and the position of the first window to the drawing module; The step of obtaining the third layer includes: in response to the second message, the drawing module obtains the third layer using the position of the first window; The method further includes: The drawing module sends a third message and the third layer to the image synthesizer; In response to the third message, the image synthesizer does not invoke the image processor GPU to synthesize the rounded corner effect of the first window; The step of compositing the fourth layer corresponding to the first window and the third layer corresponding to the third window includes: the image compositer calling the HWC to composite the fourth layer and the third layer; The second content to be displayed in the fourth layer is drawn by the drawing module or refreshed by the image synthesizer.

16. The method according to claim 15, characterized in that, The image synthesizer sends a second message and the position of the first window to the rendering module, including: If the image synthesizer does not detect a preset layer, the image synthesizer sends a second message and the position of the first window to the drawing module. The preset layer includes the layer corresponding to the game application.

17. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 16.

18. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 16.

Images