Layer composition method and apparatus

Abstract

The embodiments of the present application relate to the field of terminals. Provided are a layer composition method and apparatus, which can reduce the power consumption of an electronic device. The method comprises: in a first Vsync signal cycle, a SurfaceFlinger receiving a first buffer corresponding to a first layer; when a first preset condition is met, caching the first buffer into a preset storage space, wherein the first preset condition comprises at least one of the following: the frame rate of the first layer is less than or equal to a preset frame rate, and there are currently a plurality of layers having frame rates less than or equal to the preset frame rate; in a second Vsync signal cycle, the SurfaceFlinger receiving a second buffer corresponding to a second layer; and in the Vsync signal cycle following the second Vsync signal cycle, the SurfaceFlinger performing composition processing on the first buffer and the second buffer.

Description

A method and apparatus for layer compositing

[0001] This application claims priority to Chinese Patent Application No. 202510201870.3, filed with the State Intellectual Property Office of China on February 21, 2025, entitled "A Method and Apparatus for Layer Synthesis", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to a layer composition method and apparatus. Background Technology

[0003] With the continuous development of electronic devices, their functions are becoming increasingly diverse. For example, users can simultaneously conduct video calls using a video calling app and browse short videos using a short video app. In other words, electronic devices can display both a video call layer (shown in a small window) and a short video layer at the same time.

[0004] In some cases, electronic devices have high screen refresh rates while application layers have low frame rates. However, when multiple low-frame-rate layers exist simultaneously (e.g., video call layers and short video layers), the electronic device needs to synthesize different layers at different Vsync signal cycles because the layer data (buffer) of different layers may arrive at different vertical synchronization (Vsync) signal cycles. This results in a higher frame rate for synthesizing multiple layers, which in turn leads to higher power consumption for the electronic device. Summary of the Invention

[0005] This application provides a layer compositing method and apparatus that can reduce the power consumption of electronic devices.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, a layer compositing method is provided, applied to an electronic device, the electronic device including an image compositer SurfaceFlinger. The method includes: during a first vertical synchronization (Vsync) signal cycle, SurfaceFlinger receives first layer data corresponding to a first layer; under the condition of satisfying a first preset condition, SurfaceFlinger caches the first layer data in a preset storage space; wherein the first preset condition includes at least one of the following: the frame rate of the first layer is less than or equal to a preset frame rate, and there are currently multiple layers with frame rates less than or equal to the preset frame rate; during a second Vsync signal cycle, SurfaceFlinger receives second layer data corresponding to a second layer, the second layer being different from the first layer, and the second Vsync signal cycle is later than the first Vsync signal cycle; during the next Vsync signal cycle of the second Vsync signal cycle, SurfaceFlinger performs compositing processing on the first layer data and the second layer data.

[0008] Based on the method provided in this application, in scenarios with multiple low frame rate layers (i.e., when multiple low frame rate layers currently exist), after SurfaceFlinger receives the first layer data (with a frame rate less than or equal to a preset frame rate), it can temporarily cache the first layer data. Upon receiving the second layer data of another layer (e.g., the second layer), the cached first layer data and the second layer data of the second layer can be combined for compositing, thereby reducing the frequency at which SurfaceFlinger synthesizes multiple layers and thus reducing the power consumption of the electronic device.

[0009] In one possible implementation, the first preset condition further includes: during the first Vsync signal cycle, there is no layer data to be synthesized in the transmission queue, and / or the preset storage space does not store layer data corresponding to any layer other than the first layer. That is, if there is no layer data to be synthesized in the transmission queue, and / or the preset storage space does not store layer data corresponding to any layer other than the first layer, the first layer data can be temporarily cached. If there is layer data to be synthesized in the transmission queue, there is no need to cache the first layer data; the first layer data can be directly stored in the transmission queue so that it can be synthesized together with the existing layer data to be synthesized in the transmission queue. This eliminates the need to cache the buffer corresponding to the low frame rate layer, avoiding additional processing and reducing the power consumption of the electronic device. If there is layer data corresponding to any layer other than the first layer in the preset storage space, there is no need to cache the first layer data; the first layer data can be synthesized together with the layer data corresponding to that other layer. This eliminates the need to cache the buffer corresponding to the low frame rate layer, avoiding additional processing and reducing the power consumption of the electronic device.

[0010] In one possible implementation, the method further includes: before SurfaceFlinger caches the first layer data in a preset storage space, if the preset storage space contains the third layer data corresponding to the first layer, SurfaceFlinger performs composite processing on the third layer data in the next Vsync signal cycle after the first Vsync signal cycle. This avoids the problem of layer data accumulation in the first layer, thereby preventing stuttering in the display of the first layer.

[0011] In one possible implementation, before SurfaceFlinger performs the compositing process on the first layer data and the second layer data, the method further includes: under the condition that a second preset condition is met, SurfaceFlinger stores the second layer data in a transmission queue and retrieves the first layer data from a preset storage space and stores it in the transmission queue; wherein the second preset condition includes at least one of the following: the frame rate of the second layer is less than or equal to a preset frame rate; there are currently multiple layers with frame rates less than or equal to the preset frame rate; and the preset storage space stores layer data corresponding to other layers besides the second layer. In this way, SurfaceFlinger can perform compositing processing on the layer data (first layer data and second layer data) of two low frame rate layers (first layer and second layer) together, thereby reducing the frequency of SurfaceFlinger compositing multiple layers and thus reducing the power consumption of the electronic device.

[0012] In one possible implementation, if the first preset condition is not met, SurfaceFlinger performs compositing processing on the first layer data in the next Vsync signal cycle of the first Vsync signal cycle. That is, if the first preset condition is not met (there are no multiple low frame rate layers (layers with frame rates less than or equal to the preset frame rate), or the currently received layer data does not belong to a low frame rate layer (i.e., the first layer is not a low frame rate layer)), the first layer data can be directly composited without caching the first layer data, avoiding additional processing and reducing the power consumption of the electronic device.

[0013] In one possible implementation, SurfaceFlinger includes a layer data processing module and a variable refresh rate processing module. SurfaceFlinger receiving first layer data corresponding to a first layer includes: the layer data processing module receiving the first layer data corresponding to the first layer; the layer data processing module sending a notification message to the variable refresh rate processing module, the notification message including the identifier of the first layer; in response to the notification message, the variable refresh rate processing module incrementing the layer data count corresponding to the first layer; SurfaceFlinger caching the first layer data in a preset storage space includes: the variable refresh rate processing module determining whether the frame rate of the first layer is less than a preset frame rate; in the first layer... If the frame rate is lower than the preset frame rate, the variable refresh rate processing module determines whether there are multiple layers with frame rates less than or equal to the preset frame rate. If multiple layers with frame rates less than or equal to the preset frame rate exist, the variable refresh rate processing module determines whether layer data is stored in the preset storage space. If layer data is stored in the preset storage space, the variable refresh rate processing module determines whether the layer data stored in the preset storage space belongs to the first layer. If the layer data stored in the preset storage space does not belong to the first layer, the variable refresh rate processing module sends a first instruction message to the layer data processing module. In response to the first instruction message, the layer data processing module caches the first layer data in the preset storage space. That is, the variable refresh rate processing module can determine whether the first preset condition is met, and if the first preset condition is met, it can instruct the layer data processing module to cache the first layer data in the preset storage space. Furthermore, upon receiving second-layer data from another low-frame-rate layer (e.g., the second layer), the variable refresh rate processing module can instruct the layer data processing module to combine the cached first-layer data with the second-layer data of the second layer together to reduce the frequency at which SurfaceFlinger combines multiple layers, thereby reducing the power consumption of the electronic device.

[0014] In one possible implementation, SurfaceFlinger further includes a layer history module. The method further includes: the layer history module calculating the frame rate of multiple layers based on the timestamps corresponding to the layer data of multiple layers, including a first layer; the layer history module sending the frame rates of the multiple layers to a variable refresh rate processing module; and the variable refresh rate processing module storing information about low frame rate layers among the multiple layers, including the identifier and number of layers with frame rates less than or equal to a preset frame rate. Thus, the variable refresh rate processing module can determine whether multiple low frame rate layers exist based on the stored information about low frame rate layers, and whether the currently received layer data (e.g., first layer data) belongs to a low frame rate layer. If multiple low frame rate layers exist (i.e., multiple low frame rate layers currently exist), after SurfaceFlinger receives the first layer data (with a frame rate less than or equal to the preset frame rate), it can temporarily cache the first layer data. Upon receiving the second layer data of another low frame rate layer (e.g., a second layer), the cached first layer data and the second layer data of the second layer can be combined for compositing, reducing the frequency at which SurfaceFlinger combines multiple layers and thus reducing the power consumption of the electronic device.

[0015] In one possible implementation, the first layer and / or the second layer are video layers. Based on the method provided in the embodiments of this application, after SurfaceFlinger receives layer data (e.g., first layer data) of a low frame rate video layer (e.g., the first layer), it can temporarily cache the first layer data. After receiving layer data (second layer data) of another low frame rate layer (e.g., the second layer), the first layer data and the second layer data can be combined for compositing, thereby reducing the frequency at which SurfaceFlinger performs compositing processing on layer data of multiple layers, and thus reducing the power consumption of the electronic device.

[0016] Secondly, this application provides a chip system including one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The aforementioned chip system can be applied to electronic devices including communication modules and memory. The interface circuits are used to receive signals from the memory of the electronic device and send the received signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device can perform the methods described in the first aspect and any of its possible design embodiments.

[0017] Thirdly, this application provides a computer-readable storage medium including computer instructions. When the computer instructions are executed on an electronic device (such as a mobile phone), they cause the electronic device to perform the methods described in the first aspect and any of its possible design embodiments.

[0018] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect and any possible design thereof.

[0019] Fifthly, embodiments of this application provide a layer compositing apparatus, including a processor and a memory coupled together. The memory stores program instructions, which, when executed by the processor, cause the apparatus to implement the method described in the first aspect and any possible design of the method. The apparatus may be an electronic device or a server device; or it may be a component of an electronic device or a server device, such as a chip.

[0020] In a sixth aspect, embodiments of this application provide a layer compositing apparatus, which can be divided into different logical units or modules according to their functions, each unit or module performing different functions, so that the apparatus performs the method described in the first aspect and any of its possible design methods.

[0021] It is understood that the beneficial effects achieved by the chip system described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the apparatus described in the fifth and sixth aspects can be referred to the beneficial effects in the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0022] Figure 1 is a schematic diagram of a display provided in an embodiment of this application;

[0023] Figure 2 is a schematic diagram of buffer processing in a related technology;

[0024] Figure 3 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0025] Figure 4 is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0026] Figure 5 is a flowchart illustrating a layer compositing method provided in an embodiment of this application;

[0027] Figure 6 is a schematic diagram of inter-module interaction provided in an embodiment of this application;

[0028] Figure 7 is a schematic diagram of buffer processing provided in an embodiment of this application;

[0029] Figure 8 is a schematic diagram of another buffer processing method provided in an embodiment of this application;

[0030] Figure 9 is a schematic diagram of another buffer processing method provided in an embodiment of this application;

[0031] Figure 10 is a schematic flowchart of a method provided in an embodiment of this application;

[0032] Figure 11 is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0033] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant concepts or technologies is given first:

[0034] Currently, in the presence of multiple layers with low frame rates (below the screen refresh rate), for example, as shown in Figure 1, an electronic device can simultaneously display a video call layer (e.g., a video call window 101) and a short video layer 102. The frame rates of the video call layer and the short video layer can be lower than the screen refresh rate. Since the layer data (buffer) of different layers may arrive at different Vsync signal cycles, the electronic device needs to synthesize different layers at different Vsync signal cycles, resulting in a high frame rate when synthesizing multiple layers, which in turn leads to higher power consumption of the electronic device.

[0035] Here, "buffer" can refer to a surface object, which can include content such as images, graphics, and text. For example, a buffer corresponding to a video call layer can include one frame of image data corresponding to the video call layer, and a buffer corresponding to a short video layer can include one frame of image data corresponding to the short video layer.

[0036] In related technologies, each time the image compositer (SurfaceFlinger) receives a buffer corresponding to a layer, it can store the buffer in a transmission queue and trigger a scheduling frame so that the buffer can be composited in the next Vsync signal cycle.

[0037] For example, assuming a screen refresh rate of 60Hz, the duration of each Vsync signal cycle can be approximately 16.6ms. Taking layers A and B as examples, the frame rate of layer A (e.g., a video call layer) and layer B (e.g., a short video layer) can be 30fps. Typically, layers A and B can each generate a buffer every two Vsync signal cycles (i.e., every 33.33ms). As shown in Figure 2, within the first Vsync signal cycle, SurfaceFlinger can receive buffer A1 corresponding to layer A, store buffer A1 in a transmission queue (e.g., mTransactionQueue), and trigger a scheduling frame to perform compositing (commit and composite) on buffer A1 in the next Vsync signal cycle (the second Vsync signal cycle). After compositing buffer A1, the transmission queue can be empty. During the second Vsync signal cycle, SurfaceFlinger can receive buffer B1 corresponding to layer B, store buffer B1 in mTransactionQueue, and trigger a scheduling frame to perform composite processing on buffer B1 during the next Vsync signal cycle (the third Vsync signal cycle). After composite processing of buffer B1, the transmission queue can be empty. Similarly, during the third Vsync signal cycle, SurfaceFlinger can receive buffer A2 corresponding to layer A, store buffer A2 in mTransactionQueue, and trigger a scheduling frame to perform composite processing on buffer A2 during the next Vsync signal cycle (the fourth Vsync signal cycle). After composite processing of buffer A2, the transmission queue can be empty. During the fourth Vsync signal cycle, SurfaceFlinger can receive buffer B2 corresponding to layer B, store buffer B2 in mTransactionQueue, and trigger a scheduling frame to perform composite processing on buffer B2 during the next Vsync signal cycle (the fifth Vsync signal cycle). After the buffer B2 is processed, the transmission queue can be empty.

[0038] In summary, although all layers (e.g., layer A and layer B) correspond to a frame rate of 30fps, SurfaceFlinger actually composites multiple layers at a frequency of 60Hz. That is, SurfaceFlinger requires compositing processing for each Vsync signal cycle, resulting in a high operating frequency and consequently higher power consumption for the electronic device.

[0039] This application provides a layer compositing method that can be applied to scenarios where electronic devices simultaneously display multiple low frame rate layers (layers with frame rates lower than the screen refresh rate). This method can reduce the frequency of SurfaceFlinger compositing multiple layers, thereby reducing the power consumption of electronic devices.

[0040] Figure 3 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application.

[0041] As shown in Figure 3, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0042] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0043] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0045] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0046] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0047] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0048] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0049] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0050] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0051] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0052] Display screen 194 is used to display images, videos, etc.

[0053] Camera 193 can include 1 to N cameras. For example, an electronic device can include 2 front-facing cameras and 4 rear-facing cameras.

[0054] The methods described in the following embodiments can all be implemented in the electronic device 100 having the above-described hardware structure.

[0055] For example, the electronic device 100 may be a mobile phone, tablet computer, laptop computer, in-vehicle device, wearable device or other computing device with display function, and this application does not limit it.

[0056] The software system of the aforementioned electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 100.

[0057] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through interfaces. In some embodiments, the Android system may include an application layer, an application framework layer, a native framework layer, and a hardware layer. The Android system may also include other layers, such as a driver layer and a kernel layer, which are not limited here. It should be noted that this application uses the Android system as an example; in other operating systems (such as HarmonyOS, iOS, etc.), the solution of this application can also be implemented as long as the functions implemented by each functional module are similar to those in the embodiments of this application.

[0058] The application layer can include a series of application packages.

[0059] As shown in Figure 4, the application package may include video, games, maps, WLAN, music, SMS, gallery, calls, navigation, etc. Of course, the application layer may also include other application packages, such as Bluetooth, calendar, camera, settings, etc., which are not limited in this application.

[0060] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. These may include, for example, an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, etc., though this embodiment does not impose any limitations on them.

[0061] The native layer includes some local services and some libraries, which can be used to perform complex calculations or interact with the underlying hardware drivers.

[0062] In this embodiment, the Native layer may include SurfaceFlinger. SurfaceFlinger may include a layer data processing module, a layer module, a layer history module, and a variable refresh rate handling module, etc.

[0063] The layer data processing module is the core code module of SurfaceFlinger. It can receive layer buffers and send the layer information corresponding to the buffers (e.g., layer identifiers) to the VariableRefreshRateHandler module. In other words, it can notify the VariableRefreshRateHandler module to increase the count of the layer buffers it stores.

[0064] The Layer module can notify the VariableRefreshRateHandler module to reduce the number of layer buffers it stores when the buffers are finished being composited and the buffers of each layer are ready to be released.

[0065] The LayerHistory module can be used to record the frame rate information of each layer. In this embodiment, the LayerHistory module can collect the frame rate information (e.g., average frame rate) of each layer and send this information to VariableRefreshRateHandler.

[0066] The VariableRefreshRateHandler module is a self-developed module with the ability to handle screen refresh rate and frame rate related features. In this embodiment, the VariableRefreshRateHandler module can store the buffer count of each layer and the real-time frame rate of each layer, and can determine whether to cache the layer's buffer to a preset storage space based on the above information, temporarily skip the compositing of the buffer, and then compose the buffer together with the buffers of other layers later.

[0067] The hardware layer of an electronic device may include a central processing unit (CPU), a GPU, double data rate synchronous dynamic random-access memory (DDR SDRAM), etc. Of course, the hardware layer may also include other hardware, such as a display, a camera, etc., which are not specifically limited in this application.

[0068] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. In the description of this application, unless otherwise stated, "at least one" refers to one or more, and "more than one" refers to two or more. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0069] For ease of understanding, the layer composition method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0070] As shown in Figure 5, this application provides a layer compositing method applied to an electronic device, the electronic device including SurfaceFlinger, the method comprising:

[0071] 501. SurfaceFlinger retrieves layer-related information.

[0072] It is understandable that the display interface of an electronic device can consist of multiple surfaces. Layer-related information may include the number of buffers corresponding to one or more layers (e.g., a low frame rate layer), the count of low frame rate layers, and the identifier corresponding to the low frame rate layer. A low frame rate layer refers to a layer whose frame rate (average frame rate) is less than or equal to a preset frame rate.

[0073] For example, as shown in Figure 6, SurfaceFlinger may include a layer data processing module, a variable refresh rate processing module, a layer module, and a layer history module. The layer data processing module can receive buffers corresponding to various layers (e.g., a first layer and a second layer). The first layer and the second layer can be layers of the same application or layers of different applications. For example, the first layer can be a video call layer, and the second layer can be a short video layer. For instance, in a scenario where an electronic device simultaneously displays a video call interface (e.g., displayed in a small window) and a short video interface, the video call application can generate a buffer corresponding to the video call layer and send its drawn buffer to the layer data processing module. Similarly, the short video application can generate a buffer corresponding to the short video layer and send its drawn buffer to the layer data processing module. Each time the layer data processing module receives a buffer corresponding to a layer, it can send a notification message (a first notification message) to the variable refresh rate processing module. This notification message is used to notify that the buffer corresponding to the corresponding layer has been received. For example, the first notification message may carry the layer identifier corresponding to the buffer received by the layer data processing module. In response to this notification, the variable refresh rate processing module can update (e.g., increase) its stored buffer count (layer data count) for the corresponding layer. For example, when the layer data processing module receives the buffer corresponding to the first layer, it can send a notification to the variable refresh rate processing module.

[0074] In some embodiments, the layer data processing module can send a notification message to the variable refresh rate processing module based on the setTransactionState method. This notification message is used to notify that the buffer corresponding to the first layer has been received. The variable refresh rate processing module can increment the buffer count corresponding to the first layer by one (mLowFpsLayerBufferNumMap[layerName]++). For example, the buffer count corresponding to the first layer can be recorded as modifyLowFpsLayerBufferNumMap(layerName, 1).

[0075] After buffer compositing is complete, the layer module can send a notification message (second notification message) to the variable refresh rate processing module to instruct the variable refresh rate processing module to reduce the buffer count corresponding to the corresponding layer it stores. The second notification message may carry the layer identifier corresponding to the composed buffer. In some embodiments, the layer module can send the layer identifier corresponding to the composed buffer to the variable refresh rate processing module based on the `decrementPendingBufferCount` method. After receiving the notification message sent by the layer module, the variable refresh rate processing module can reduce the buffer count corresponding to the corresponding layer it stores (`mLowFpsLayerBufferNumMap[layerName]--`). For example, the buffer count corresponding to the first layer can be recorded as `modifyLowFpsLayerBufferNumMap(layerName, -1)`.

[0076] The layer history module can save the timestamp of each buffer corresponding to each layer and calculate the frame rate (average frame rate) of each layer based on the timestamp of each buffer. Furthermore, the layer history module can send the average frame rate of each layer to the variable refresh rate processing module. In some embodiments, the layer history module can calculate the average frame rate of each layer using the `summarize` function. The average frame rate can be the average of the frame rates calculated multiple times for the same layer. The variable refresh rate processing module can receive the average frame rate of each layer from the layer history module (`addInfoLowFpsLayer(layerName, fps)`). If the average frame rate of a layer is less than or equal to a preset frame rate (e.g., <45), the variable refresh rate processing module can mark this layer as a low frame rate layer (`mCurrentLowFpsLayer(layerName) = fps`) and increment the low frame rate layer count (`InfoLowFpsLayerNum++`). The low frame rate layer count indicates the number of low frame rate layers. The preset frame rate can be less than or equal to the screen refresh rate. For example, the screen refresh rate can be 60Hz, and the preset frame rate can be 45fps.

[0077] Furthermore, the variable refresh rate processing module can determine whether to cache the buffers of each layer based on the acquired layer-related information, and can send the determination result to the layer data processing module. The layer data processing module can then perform corresponding processing on the buffers based on the determination result. For example, if the determination result indicates that the buffers should be cached, the layer data processing module can perform caching processing on the buffers. If the determination result indicates that the buffers should not be cached, the layer data processing module can store the buffers in a transmission queue so that they can be composited when the next Vsync signal arrives.

[0078] The following section uses the first buffer of the first layer as an example to explain how SurfaceFlinger processes the first buffer based on the acquired layer information.

[0079] 502. SurfaceFlinger receives the first buffer of the first layer.

[0080] SurfaceFlinger can receive the first buffer of the first layer through the layer data processing module.

[0081] In some embodiments, the layer data processing module may receive the buffer (e.g., the first buffer) of the corresponding layer (e.g., the first layer) sent by the application (e.g., the first application) based on the setTransactionState method.

[0082] 503. SurfaceFlinger determines whether the first layer is a low frame rate layer.

[0083] After receiving the first buffer of the first layer, the SurfaceFlinger layer data processing module can send a notification message to the variable refresh rate processing module (to notify that the first buffer of the first layer has been received). After receiving the notification message, the variable refresh rate processing module can determine whether the first layer is a low frame rate layer.

[0084] Understandably, the variable refresh rate processing module can store and record low frame rate layers (e.g., store the identifier corresponding to the low frame rate layer). Refer to step 501 for related explanations. The variable refresh rate processing module can compare the identifier corresponding to the first layer with the identifiers of the low frame rate layers it stores. If the identifier corresponding to the first layer matches the identifier of the low frame rate layer it stores (i.e., the identifier corresponding to the first layer belongs to one of the identifiers of the low frame rate layers it stores), the variable refresh rate processing module determines that the first layer is a low frame rate layer.

[0085] If the first layer is a low frame rate layer, step 504 can be executed.

[0086] If the first layer is not a low frame rate layer, step 510 can be performed.

[0087] 504. SurfaceFlinger determines whether there are multiple low frame rate layers.

[0088] The existence of multiple low frame rate layers can be determined based on the count of low frame rate layers stored in the variable refresh rate processing module. If the count of low frame rate layers stored in the variable refresh rate processing module is greater than or equal to 2, it indicates that multiple low frame rate layers exist.

[0089] If multiple low frame rate layers exist, step 505 can be executed. If no multiple low frame rate layers exist (e.g., the low frame rate layer count is less than 2), step 510 can be executed.

[0090] 505. SurfaceFlinger determines whether there is a buffer to be synthesized in the transmission queue during the current Vsync signal cycle.

[0091] The current Vsync signal period (e.g., the first Vsync signal period) can refer to the Vsync signal period during which the layer data processing module receives the first buffer of the first layer. The transmission queue can be used to store buffers to be composited. Buffers in the transmission queue can be composited in the next Vsync signal period to avoid stuttering in high frame rate layers. A high frame rate layer can refer to a layer with a frame rate greater than a preset frame rate.

[0092] In some embodiments, SurfaceFlinger can receive buffers from layers other than the first layer (e.g., the third layer) during the current Vsync signal cycle. After receiving the buffer corresponding to the third layer, SurfaceFlinger can determine the frame rate of the third layer. If the third layer is a high frame rate layer, its buffer can be directly sent to the transmission queue. In this case, there are buffers to be synthesized in the transmission queue during the current Vsync signal cycle.

[0093] Additionally, if the third layer is not a high frame rate layer, the processing of the buffer for the third layer can refer to the processing of the buffer for the first layer, which will not be elaborated here.

[0094] In some embodiments, SurfaceFlinger can query a first variable (e.g., mAllowToSkip) that indicates whether a buffer to be composed exists in the transport queue. For example, mAllowToSkip = true indicates that there is no buffer to be composed in the transport queue. mAllowToSkip = false indicates that there is a buffer to be composed in the transport queue. For instance, after SurfaceFlinger sends a buffer for a high frame rate layer (e.g., the third layer) to the transport queue, it can set mAllowToSkip = false. It is understood that since the buffer in the transport queue can be cleared after each compositing process by SurfaceFlinger, mAllowToSkip can be reset to true after each compositing process by SurfaceFlinger.

[0095] If there is no buffer to be synthesized in the transmission queue during the current Vsync signal cycle, step 506 can be executed, that is, to determine whether the low frame rate layer has a cached buffer in the preset storage space.

[0096] If there is a buffer to be synthesized in the transmission queue during the current Vsync signal cycle, step 510 can be executed.

[0097] 506. SurfaceFlinger determines whether there is a cached buffer in the preset storage space.

[0098] In this embodiment, a preset storage space is used to cache the buffer corresponding to the layer. The buffer cached in the preset storage space can be delayed in the compositing process.

[0099] In some embodiments, the buffer (e.g., buffer n) corresponding to a certain layer (e.g., layer A) can be cached in a preset storage space if the following conditions are met: the corresponding layer is a low frame rate layer, and multiple low frame rate layers currently exist. Optionally, it can also be determined whether the preset storage space stores buffers of other layers (layers other than layer A) during the current Vsync signal period, and / or whether there is a buffer to be synthesized in the transmission queue. If the preset storage space does not store buffers of other layers (layers other than layer A) during the current Vsync signal period, and there is no buffer to be synthesized in the transmission queue, the buffer corresponding to the corresponding layer (layer A) can be cached in the preset storage space.

[0100] If there is a cached buffer in the preset storage space, step 507 can be executed, which determines whether the cached buffer in the preset storage space belongs to the first layer. If there is no cached buffer in the preset storage space, step 508 can be executed.

[0101] 507. SurfaceFlinger determines whether a cached buffer in the preset storage space belongs to the first layer.

[0102] In some scenarios, SurfaceFlinger may receive multiple buffers corresponding to the first layer consecutively. For example, before receiving the first buffer corresponding to the first layer, SurfaceFlinger may also receive other buffers corresponding to the first layer (e.g., a third buffer). SurfaceFlinger can cache the third buffer in a preset storage space. In this case, the buffers already cached in the preset storage space (e.g., the third buffer) belong to the first layer.

[0103] If the buffer already cached in the preset storage space belongs to the first layer, step 509 can be executed.

[0104] If the buffer already cached in the preset storage space does not belong to the first layer, step 510 can be executed.

[0105] 508. SurfaceFlinger caches the first buffer to the preset storage space.

[0106] In other words, if there are no cached buffers in the preset storage space, SurfaceFlinger can cache the first buffer in the preset storage space. In this case, the preset storage space includes the first buffer but does not include other buffers.

[0107] In some embodiments, the variable refresh rate processing module of SurfaceFlinger can determine whether to cache the buffers of each layer based on the acquired layer-related information, and can send the determination result to the layer data processing module. The layer data processing module can perform corresponding processing on the buffers according to the determination result. For example, the determination result can be indicated by a second variable (e.g., SkipCurrent).

[0108] In some embodiments, after step 506, if there is no cached buffer in the preset storage space, the variable refresh rate processing module can set SkipCurrent = 1 and notify the layer data processing module. When the layer data processing module determines that SkipCurrent = 1, it can cache the first buffer in the preset storage space. Thus, when the buffer for the next low frame rate layer (e.g., the second layer) arrives (i.e., after the layer data processing module receives the buffer for the next low frame rate layer (e.g., the second layer), the first and second buffers can be combined for compositing, reducing the frequency at which SurfaceFlinger composes multiple layers and thereby reducing the power consumption of the electronic device.

[0109] For example, assume a screen refresh rate of 60Hz, meaning each Vsync signal cycle can be 16.6ms. Layer A (the first layer, e.g., the video call layer) and Layer B (the second layer, e.g., the short video layer) can have a frame rate of 30fps. That is, Layer A and Layer B can each generate a buffer every two Vsync signal cycles (i.e., every 33.33ms). As shown in Figure 7, in the first Vsync signal cycle, SurfaceFlinger can receive buffer A1 (the first buffer) corresponding to Layer A. After receiving buffer A1, SurfaceFlinger can cache the first buffer in a preset storage space (e.g., mPendingTransaction). In the second Vsync signal cycle, SurfaceFlinger can receive buffer B1 (the second buffer) corresponding to Layer B. After receiving buffer B1, SurfaceFlinger finds that buffer A1 is stored in the preset storage space. It can then retrieve buffer A1 from the preset storage space and store both buffer A1 and buffer B1 into the transmission queue (e.g., mTransactionQueue), triggering a scheduling frame. In the next Vsync signal cycle (the third Vsync signal cycle), SurfaceFlinger can perform a composite process (commit and composite) on buffer A1 and buffer B1. After this process, the transmission queue can be empty. Similarly, in the third Vsync signal cycle, SurfaceFlinger can receive buffer A2 corresponding to layer A. After receiving buffer A2, SurfaceFlinger can cache it in the preset storage space (mPendingTransaction). In the fourth Vsync signal cycle, SurfaceFlinger can receive buffer B2 corresponding to layer B. After SurfaceFlinger receives buffer B2, it finds that buffer A2 is stored in the preset storage space. It can then retrieve buffer A2 from the preset storage space, store both buffer A2 and buffer B2 in the transmission queue (e.g., mTransactionQueue), and trigger a scheduling frame.Thus, in the next Vsync signal cycle (the fifth Vsync signal cycle), SurfaceFlinger can perform commit and composite processing on buffer A2 and buffer B2. After the composite processing of buffer A2 and buffer B2, the transfer queue can be empty.

[0110] This allows the buffers of two layers (e.g., layer A and layer B) to be composited together, reducing the frequency of SurfaceFlinger compositing multiple layers and thus reducing the power consumption of electronic devices.

[0111] 509. SurfaceFlinger caches the first buffer to a preset storage space and stores the cached buffer into a transmission queue, then performs synthesis processing on the buffers in the transmission queue.

[0112] In some embodiments, if the buffer (e.g., the third buffer) cached in the preset storage space belongs to the first layer (e.g., layer A is the first layer), the variable refresh rate processing module can set SkipCurrent=2 and notify the layer data processing module. When the layer data processing module determines that SkipCurrent=2, it can cache the first buffer in the preset storage space. This allows the first and second buffers to be combined together when the buffer (e.g., the second buffer) of the next low frame rate layer (e.g., the second layer) arrives (i.e., after the layer data processing module receives the second buffer of the next low frame rate layer (e.g., the second layer), reducing the frequency of SurfaceFlinger combining multiple layers. Furthermore, the layer data processing module can store the cached buffer (e.g., the third buffer) in the transmission queue so that the cached buffer (e.g., buffer n) can be combined in the next Vsync signal cycle of the current Vsync signal cycle. This avoids the problem of buffer accumulation in the first layer, thereby preventing stuttering in the display of the first layer.

[0113] As shown in Figure 8, taking a screen refresh rate of 60Hz (i.e., each Vsync signal cycle can be 16.6ms) and a frame rate of 30fps for layer A (the first layer, e.g., the video call layer) and layer B (the second layer, e.g., the short video layer) as an example, within the third Vsync signal cycle, SurfaceFlinger can first receive buffer A2 (the third buffer) corresponding to layer A, and then receive buffer A3 (the first buffer) corresponding to layer A. After receiving buffer A2 corresponding to layer A, SurfaceFlinger can cache buffer A2 into a preset storage space (mPendingTransaction). After receiving buffer A3 corresponding to layer A, SurfaceFlinger can cache buffer A3 into the preset storage space (mPendingTransaction), and retrieve the cached buffer A2 from the preset storage space (mPendingTransaction), store it in a transmission queue (e.g., mTransactionQueue), and then trigger a scheduling frame so that SurfaceFlinger can perform composite processing on buffer A2. After composite processing on buffer A2, the transmission queue can be empty. During the fourth Vsync signal cycle, SurfaceFlinger can receive buffer B2 (the second buffer) corresponding to layer B. After receiving buffer B2, SurfaceFlinger finds that buffer A3 (the first buffer) is stored in the preset storage space. It can then retrieve buffer A3 from the preset storage space and store both buffer A3 and buffer B2 into the transmission queue (e.g., mTransactionQueue), triggering a scheduling frame. Thus, during the next Vsync signal cycle (the fifth Vsync signal cycle), SurfaceFlinger can perform a composite process (commit and composite) on buffer A3 and buffer B2. After the composite process, the transmission queue can be empty.

[0114] This not only avoids the problem of buffer accumulation in layer A, but also allows the buffer of layer A to be composited together with the buffer of the next low frame rate layer (e.g., layer B) (e.g., the second buffer), thereby reducing the frequency of SurfaceFlinger compositing multiple layers and thus reducing the power consumption of electronic devices.

[0115] 510. SurfaceFlinger does not cache the first buffer, but stores the first buffer in the transfer queue and performs synthesis processing on the buffers in the transfer queue.

[0116] In other words, if the first layer is not a low frame rate layer, or if there are no multiple low frame rate layers at present, or if there is a buffer to be composed in the transmission queue during the current Vsync signal cycle, SurfaceFlinger will not cache the first buffer, but will store the first buffer in the transmission queue for compositing.

[0117] In some embodiments, if the first layer is not a low frame rate layer, or if there are no multiple low frame rate layers, or if there is a buffer to be composited in the transmission queue during the current Vsync signal period, the variable refresh rate processing module can set SkipCurrent = 0 and notify the layer data processing module. If the layer data processing module determines that SkipCurrent = 0, it will not skip the first buffer (i.e., will not cache the first buffer in the preset storage space) and can store the first buffer in the transmission queue so that composited processing can be performed on the first buffer when the next Vsync signal arrives.

[0118] In some embodiments, if a cached buffer exists in the preset storage space, the layer data processing module can store both the first buffer and the cached buffer into the transmission queue and perform synthesis processing on the buffers in the transmission queue.

[0119] In some embodiments, after the layer data processing module stores the first buffer into the transmission queue, it can set mAllowToSkip = false, indicating that there are buffers to be synthesized in the transmission queue. In this way, after receiving other buffers within the current Vsync signal cycle, the layer data processing module can determine whether there are buffers to be synthesized in the transmission queue based on mAllowToSkip.

[0120] As shown in Figure 9, taking a screen refresh rate of 60Hz (i.e., each Vsync signal cycle can be 16.6ms), and layer A (the first layer, e.g., the video call layer) and layer B (the second layer, e.g., the short video layer) with a frame rate of 30fps, and layer C with a frame rate of 60fps as an example. Layers A and B are low frame rate layers, and layer C is a high frame rate layer. In the first Vsync signal cycle, SurfaceFlinger can first receive buffer C1 corresponding to layer C. Since layer C is a high frame rate layer, SurfaceFlinger can directly store buffer C1 into the transmission queue (e.g., mTransactionQueue). In the first Vsync signal cycle, SurfaceFlinger can also receive buffer A1 (the first buffer) corresponding to layer A. Although layer A is a low frame rate layer, and there are currently multiple (e.g., two) low frame rate layers (layer A and layer B). However, since a buffer (buffer C1) exists in the transmission queue, SurfaceFlinger can store buffer A1 in the transmission queue without skipping it (i.e., without caching buffer A1 to the preset storage space). Thus, in the next Vsync signal cycle (the second Vsync signal cycle), SurfaceFlinger can perform a composite process (commit and composite) on buffer A1 and buffer C1. After the composite process, the transmission queue can be empty. Similarly, in the second Vsync signal cycle, SurfaceFlinger can first receive buffer C2 corresponding to layer C. Since layer C is a high frame rate layer, SurfaceFlinger can directly store buffer C2 in the transmission queue (e.g., mTransactionQueue). In the second Vsync signal cycle, SurfaceFlinger can also receive buffer B1 corresponding to layer B. Although layer B is a low frame rate layer, and there are currently multiple (e.g., two) low frame rate layers (layer A and layer B). However, since buffer (buffer C2) exists in the transmission queue, SurfaceFlinger can store buffer B1 in the transmission queue without skipping it (i.e., without caching buffer B1 to the preset storage space). Thus, in the next Vsync signal cycle (the third Vsync signal cycle), SurfaceFlinger can perform compositing processing on buffer B1 and buffer C2.After combining buffer B1 and buffer C2, the transmission queue can be empty.

[0121] Similarly, in the third Vsync signal cycle, SurfaceFlinger can first receive buffer C3 corresponding to layer C. Since layer C is a high frame rate layer, SurfaceFlinger can directly store buffer C3 into the transfer queue (e.g., mTransactionQueue). In the third Vsync signal cycle, SurfaceFlinger can also receive buffer A2 (the first buffer) corresponding to layer A. Although layer A is a low frame rate layer, and multiple (e.g., two) low frame rate layers (layer A and layer B) currently exist, since buffer C3 is present in the transfer queue, SurfaceFlinger can store buffer A2 into the transfer queue without skipping it (i.e., without caching buffer A2 to the preset storage space). Thus, in the next Vsync signal cycle (the fourth Vsync signal cycle), SurfaceFlinger can perform a composite process (commit and composite) on buffer A2 and buffer C3. After the composite process on buffer A2 and buffer C3, the transfer queue can be empty. During the fourth Vsync signal cycle, SurfaceFlinger can first receive buffer C4 corresponding to layer C. Since layer C is a high frame rate layer, SurfaceFlinger can directly store buffer C4 into the transfer queue (e.g., mTransactionQueue). During the fourth Vsync signal cycle, SurfaceFlinger can also receive buffer B2 corresponding to layer B. Although layer B is a low frame rate layer, and there are currently multiple (e.g., two) low frame rate layers (layer A and layer B), since buffer C4 is present in the transfer queue, SurfaceFlinger can store buffer B2 into the transfer queue without skipping it (i.e., without caching buffer B2 to the preset storage space). Thus, during the next Vsync signal cycle (the fifth Vsync signal cycle), SurfaceFlinger can perform composite processing on buffer B2 and buffer C4. After composite processing of buffer B2 and buffer C4, the transfer queue can be empty.

[0122] This allows the buffers corresponding to low frame rate layers and high frame rate layers to be combined into one process, eliminating the need to cache the buffer corresponding to the low frame rate layer, avoiding extra processing operations, and reducing the power consumption of electronic devices.

[0123] It should be noted that there is no necessary execution order between steps 501 and 510, and this embodiment does not make specific restrictions on the execution order between the steps.

[0124] Based on the method provided in this application, in scenarios where multiple low frame rate layers exist (i.e., when multiple low frame rate layers currently exist), after SurfaceFlinger receives a buffer for one low frame rate layer, it can temporarily cache the buffer. After receiving a buffer for another low frame rate layer, the cached buffer can be combined with the buffer for the other low frame rate layer for compositing, thereby reducing the frequency at which SurfaceFlinger composes multiple layers and thus reducing the power consumption of the electronic device.

[0125] As shown in Figure 10, this application embodiment provides a layer compositing method applied to an electronic device, which includes SurfaceFlinger. This application embodiment uses a first buffer of a first layer and a second buffer of a second layer as examples to illustrate how SurfaceFlinger processes the first and second buffers. The method includes:

[0126] 1001. During the first Vsync signal cycle, SurfaceFlinger receives the first layer data corresponding to the first layer.

[0127] For example, the first layer data can be a first buffer, and the first buffer can be described with reference to the relevant description of the embodiment shown in Figure 5.

[0128] 1002. Under the condition that the first preset condition is met, SurfaceFlinger caches the data of the first layer to the preset storage space. The first preset condition includes at least one of the following: the frame rate of the first layer is less than or equal to the preset frame rate, and there are currently multiple layers with frame rates less than or equal to the preset frame rate.

[0129] In some embodiments, the first preset condition further includes: during the first Vsync signal period, there is no layer data to be synthesized in the transmission queue, and the preset storage space does not store layer data corresponding to other layers (low frame rate layers) besides the first layer.

[0130] In other words, after SurfaceFlinger receives the data for the first layer, if it determines that the first layer is a low frame rate layer, and there are multiple low frame rate layers (i.e., multiple layers with frame rates less than or equal to a preset frame rate), and the preset storage space does not store layer data corresponding to other layers (low frame rate layers) besides the first layer, SurfaceFlinger can temporarily cache the first layer data. Subsequently, after SurfaceFlinger receives a buffer for another layer (e.g., the second layer), it can combine the cached buffer with the buffer for the other layer to reduce the frequency of SurfaceFlinger combining multiple layers, thereby reducing the power consumption of the electronic device.

[0131] In some embodiments, SurfaceFlinger includes a layer data processing module and a variable refresh rate processing module. SurfaceFlinger receiving first layer data corresponding to a first layer includes: the layer data processing module receiving the first layer data corresponding to the first layer; the layer data processing module sending a first notification message to the variable refresh rate processing module, the first notification message including the identifier of the first layer; in response to the first notification message, the variable refresh rate processing module can increment the layer data count corresponding to the first layer and trigger a judgment process (i.e., determining whether a first preset condition is met), thereby determining whether to cache the first layer data corresponding to the first layer. The judgment process of the variable refresh rate processing module can be referred to the relevant descriptions of steps 503-507.

[0132] In some embodiments, if there is layer data to be synthesized in the transmission queue, there is no need to cache the first layer data. The first layer data can be directly stored in the transmission queue so that the first layer data can be synthesized together with the original layer data to be synthesized in the transmission queue. There is no need to cache the buffer corresponding to the low frame rate layer, avoiding extra operation processing and reducing the power consumption of electronic devices.

[0133] In some embodiments, before SurfaceFlinger caches the first layer data to a preset storage space, if the preset storage space contains the third layer data corresponding to the first layer, SurfaceFlinger performs composite processing on the third layer data in the next Vsync signal cycle after the first Vsync signal cycle. This avoids the problem of buffer accumulation for the first layer, thereby preventing stuttering in the display of the first layer.

[0134] For example, the third layer data can be a third buffer, and the third buffer can be referred to the relevant description of the embodiment shown in Figure 5.

[0135] In some embodiments, the frame rate of the second layer (e.g., layer B in the embodiment shown in FIG5) is less than or equal to a preset frame rate, and the plurality of layers with frame rates less than or equal to the preset frame rate include the first layer and the second layer.

[0136] In other embodiments, the frame rate of the second layer (e.g., layer C in the embodiment shown in FIG5) is greater than a preset frame rate, and the multiple layers with frame rates less than or equal to the preset frame rate include the first layer and the third layer (e.g., layer B in the embodiment shown in FIG5).

[0137] In some embodiments, caching first layer data in a preset storage space by SurfaceFlinger includes: a variable refresh rate processing module determining whether the frame rate of the first layer is less than a preset frame rate; if the frame rate of the first layer is less than the preset frame rate, the variable refresh rate processing module determining whether there are multiple layers with frame rates less than or equal to the preset frame rate; if there are multiple layers with frame rates less than or equal to the preset frame rate, the variable refresh rate processing module determining whether layer data is stored in the preset storage space; if layer data is stored in the preset storage space, the variable refresh rate processing module determining whether the layer data stored in the preset storage space belongs to the first layer; if the layer data stored in the preset storage space does not belong to the first layer, the variable refresh rate processing module sending a first indication message to the layer data processing module; and in response to the first indication message, the layer data processing module caching the first layer data in the preset storage space.

[0138] In some embodiments, SurfaceFlinger further includes a layer history module, and the variable refresh rate processing module can obtain the frame rate of at least one layer from the layer history module. The layer history module calculates the frame rate of multiple layers based on the timestamps corresponding to the layer data of multiple layers, including a first layer; the layer history module sends the frame rates of multiple layers to the variable refresh rate processing module; the variable refresh rate processing module stores information about low frame rate layers among the multiple layers, including the identifier and number of layers with frame rates less than or equal to a preset frame rate.

[0139] Furthermore, if the first preset condition is not met, SurfaceFlinger performs compositing processing on the first layer data in the next Vsync signal cycle following the first Vsync signal cycle. That is, if the first layer is not a low frame rate layer, or if there are no multiple low frame rate layers at present, or if the preset storage space stores layer data corresponding to other layers (low frame rate layers) besides the first layer, or if there is a buffer to be composed in the transmission queue during the current Vsync signal cycle, SurfaceFlinger does not cache the first buffer, but stores the first buffer in the transmission queue for compositing processing.

[0140] In some embodiments, if the first preset condition is not met, SurfaceFlinger can perform compositing processing on the first layer data and other layer data to be composited. The other layer data to be composited includes layer data in the transfer queue and / or preset storage space.

[0141] 1003. During the second Vsync signal cycle, SurfaceFlinger receives the second layer data corresponding to the second layer. The second layer is different from the first layer. The second Vsync signal cycle is later than the first Vsync signal cycle.

[0142] After receiving the second layer data corresponding to the second layer, SurfaceFlinger can determine whether a second preset condition is met. The second preset condition includes at least one of the following: the frame rate of the second layer is less than or equal to a preset frame rate; multiple layers currently exist with frame rates less than or equal to the preset frame rate; and the preset storage space stores layer data corresponding to other layers besides the second layer (for example, the preset storage space stores the first layer data corresponding to the first layer). If the second preset condition is met, there is no need to cache the second layer data corresponding to the second layer.

[0143] The processing flow of SurfaceFlinger after receiving the second layer data corresponding to the second layer can be referred to in the embodiment shown in Figure 5, which shows the processing flow of SurfaceFlinger after receiving the first layer data corresponding to the first layer. The difference lies in the different judgment results of SurfaceFlinger for the first layer data and the second layer data. For example, the judgment result of SurfaceFlinger for the first layer data can be indicated by SkipCurrent=1, and the judgment result of SurfaceFlinger for the second layer data can be indicated by SkipCurrent=0. For a description of SkipCurrent, please refer to the embodiment shown in Figure 5, which will not be repeated here. For example, the second layer data can be a second buffer, and the description of the second buffer can be referred to in the embodiment shown in Figure 5.

[0144] 1004. In the next Vsync signal cycle of the second Vsync signal cycle, SurfaceFlinger performs composite processing on the first layer data and the second layer data.

[0145] After receiving the second layer data, SurfaceFlinger, under the condition that a second preset condition is met, can store the second layer data in the transmission queue and retrieve the first layer data from the preset storage space and store it in the transmission queue. This allows the first and second layer data to be combined and processed together in the next Vsync signal cycle of the second Vsync signal cycle. This reduces the frequency at which SurfaceFlinger combines multiple layers (first and second layers), thereby reducing the power consumption of the electronic device.

[0146] In some embodiments, the first layer and / or the second layer are video layers. For example, the first layer may be a layer corresponding to a first video stream, in which case the data of the first layer is a frame of image data corresponding to the first video stream. The first video stream may be, for example, a short video / live video / long video stream. Similarly, the second layer may be a layer corresponding to a second video stream, in which case the data of the second layer is a frame of image data corresponding to the second video stream. The second video stream may be, for example, a video stream from a video call window / video conferencing window. It should be noted that the video layer buffer can be drawn independently of the Vsync signal. After the drawing thread of the video application obtains the video stream data, it can draw the video layer buffer and send the video layer buffer to SurfaceFlinger. Based on the method provided in this application embodiment, after SurfaceFlinger receives the buffer of a low frame rate video layer (e.g., the first layer), it can temporarily cache the buffer. After receiving a buffer for another low frame rate layer (e.g., a second layer), the buffer can be combined with the buffer for another low frame rate layer to reduce the frequency at which SurfaceFlinger combines multiple layers, thereby reducing the power consumption of electronic devices.

[0147] Based on the method provided in this application embodiment, in a scenario where multiple low frame rate layers exist (i.e., when multiple low frame rate layers currently exist), after SurfaceFlinger receives a buffer for a low frame rate layer (e.g., the first layer), it can temporarily cache the buffer. Upon receiving a buffer for another low frame rate layer (e.g., the second layer), the cached buffer can be combined with the buffer for the other low frame rate layer for composite processing. This reduces the frequency at which SurfaceFlinger composites multiple layers (i.e., reduces the frequency at which SurfaceFlinger performs composite processing on buffers for multiple layers), thereby reducing the power consumption of the electronic device.

[0148] Some embodiments of this application provide an electronic device that may include a touchscreen, a memory, and one or more processors. The touchscreen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the electronic device in the above method embodiments. The structure of this electronic device can be referred to as the structure of the electronic device 100 shown in FIG3.

[0149] This application also provides a chip system, as shown in FIG11, which includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 are interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1102 can be used to send signals to other devices (e.g., the processor 1101).

[0150] For example, interface circuit 1102 can read instructions stored in the memory of the electronic device and send the instructions to processor 1101. When the instructions are executed by processor 1101, the electronic device (electronic device 100 as shown in FIG3) can perform the various steps in the above embodiments.

[0151] Of course, the chip system may also include other discrete components, and this application embodiment does not specifically limit this.

[0152] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device (e.g., a mobile phone) in the above method embodiments.

[0153] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform various functions or steps performed by the electronic device (e.g., a mobile phone) in the above method embodiments.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A layer compositing method, characterized in that, Applied to an electronic device, the electronic device including an image synthesizer SurfaceFlinger, the method includes: During the first vertical synchronization (Vsync) signal cycle, the SurfaceFlinger receives the first layer data corresponding to the first layer; Under the condition that the first preset condition is met, SurfaceFlinger caches the first layer data to a preset storage space; wherein, the first preset condition includes at least one of the following: the frame rate of the first layer is less than or equal to a preset frame rate, and there are currently multiple layers with frame rates less than or equal to the preset frame rate; During the second Vsync signal cycle, the SurfaceFlinger receives the second layer data corresponding to the second layer, which is different from the first layer. The second Vsync signal cycle is later than the first Vsync signal cycle. In the next Vsync signal cycle of the second Vsync signal cycle, the SurfaceFlinger performs a composite processing on the first layer data and the second layer data.

2. The method according to claim 1, characterized in that, The first preset condition also includes: During the first Vsync signal cycle, there is no layer data to be synthesized in the transmission queue, and / or The preset storage space does not store layer data corresponding to any layers other than the first layer.

3. The method according to claim 1, characterized in that, The method further includes: Before SurfaceFlinger caches the first layer data into a preset storage space, if the preset storage space stores the third layer data corresponding to the first layer, SurfaceFlinger performs synthesis processing on the third layer data in the next Vsync signal cycle of the first Vsync signal cycle.

4. The method according to claim 1, characterized in that, Before SurfaceFlinger performs the compositing process on the first layer data and the second layer data, the method further includes: Under the condition that the second preset condition is met, SurfaceFlinger stores the second layer data into the transmission queue and retrieves the first layer data from the preset storage space and stores it into the transmission queue. The second preset condition includes at least one of the following: the frame rate of the second layer is less than or equal to the preset frame rate, there are currently multiple layers with frame rates less than or equal to the preset frame rate, and the preset storage space stores layer data corresponding to other layers besides the second layer.

5. The method according to claim 1, characterized in that, If the first preset condition is not met, the SurfaceFlinger performs composite processing on the first layer data in the next Vsync signal cycle of the first Vsync signal cycle.

6. The method according to any one of claims 1-5, characterized in that, The SurfaceFlinger includes a layer data processing module and a variable refresh rate processing module. The SurfaceFlinger receives the first layer data corresponding to the first layer, including: The layer data processing module receives the first layer data corresponding to the first layer; The layer data processing module sends a notification message to the variable refresh rate processing module, the notification message including the identifier of the first layer; In response to the notification message, the variable refresh rate processing module increases the layer data count corresponding to the first layer; The SurfaceFlinger caches the first layer data to a preset storage space, including: The variable refresh rate processing module determines whether the frame rate of the first layer is less than the preset frame rate; If the frame rate of the first layer is less than the preset frame rate, the variable refresh rate processing module determines whether there are multiple layers with frame rates less than or equal to the preset frame rate. When there are multiple layers with frame rates less than or equal to a preset frame rate, the variable refresh rate processing module determines whether layer data is stored in the preset storage space. When layer data is stored in the preset storage space, the variable refresh rate processing module determines whether the layer data stored in the preset storage space belongs to the first layer; If the layer data stored in the preset storage space does not belong to the first layer, the variable refresh rate processing module sends a first indication message to the layer data processing module. In response to the first indication information, the layer data processing module caches the first layer data in the preset storage space.

7. The method according to claim 6, characterized in that, The SurfaceFlinger also includes a layer history module, and the method further includes: The layer history module calculates the frame rate of the multiple layers based on the timestamps corresponding to the layer data of the multiple layers, the multiple layers including the first layer; The layer history module sends the frame rates of the multiple layers to the variable refresh rate processing module; The variable refresh rate processing module stores information about low frame rate layers among the multiple layers. The information about low frame rate layers includes the identifier and number of layers with a frame rate less than or equal to a preset frame rate.

8. The method according to claim 1, characterized in that, The first layer and / or the second layer are video layers.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a processor, implements the method as described in any one of claims 1-8.

10. An electronic device, characterized in that, The electronic device includes a processor for running a computer program stored in a memory to enable the electronic device to perform the method as described in any one of claims 1-8.

Images