Frame rate control method, electronic device, storage medium, and program product

Abstract

The present application relates to the technical field of frame rate control, and discloses a frame rate control method, an electronic device, a storage medium, and a program product. The method is applied to a first electronic device comprising a first camera. The first camera captures M video frames at a first frame rate, and then the first electronic device determines a content change degree of the M video frames. If the content change degree is greater than a first change degree threshold, the first camera captures N video frames at a second frame rate greater than the first frame rate corresponding to the M video frames, and the first electronic device displays the N video frames at the second frame rate. If the content change degree is less than a second change degree threshold, the first camera captures N video frames at a second frame rate less than the first frame rate, and the first electronic device displays the N video frames at the second frame rate. Such a method can ensure that the capture frame rate and display frame rate of N video frames are dynamically adjusted, thereby improving the display quality of the video frames, or reducing capture power consumption corresponding to the video frames.

Description

Frame rate control methods, electronic devices, storage media, and application products

[0001] This application claims priority to Chinese Patent Application No. 202411570839.9, filed on November 5, 2024, entitled "Frame Rate Control Method, Electronic Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of frame rate control technology, and in particular to a frame rate control method, electronic device, storage medium, and program product. Background Technology

[0003] When users engage in real-time video communication (such as video calls) using electronic devices, the device's camera captures video frames (including images of people, landscapes, etc.) at a certain frame rate. The electronic device then displays these captured video frames on its screen at that frame rate. It's understandable that the frame rate at which the video frames are captured affects power consumption; for example, a higher frame rate results in higher power consumption. The display frame rate (frames per second, FPS) affects display quality, which in turn impacts the user experience.

[0004] Figure 1 illustrates the relationship between user experience quality and display frame rate when making video calls using an electronic device. As shown in Figure 1, the horizontal axis represents the display frame rate, and the vertical axis represents the experience quality. It can be seen that when the display frame rate is between 0 FPS and 24 FPS, the experience quality increases with increasing display frame rate. However, after the display frame rate exceeds 24 FPS, the experience quality tends to stabilize with further increases in display frame rate. Increasing the display frame rate further at this point does not significantly improve the experience quality. Furthermore, increasing the display frame rate requires the camera to increase its capture frame rate, which increases power consumption. Therefore, in some solutions, the camera captures video frames at a fixed capture frame rate, and the electronic device displays the captured video frames at the same display frame rate (e.g., 15 FPS) to ensure high experience quality and low power consumption.

[0005] However, the required frame rate for camera capture varies across different application scenarios, and users also have different requirements for display frame rate. For example, in situations where the frame rate requirement is low (such as during video calls, where the motion of image content between adjacent frames is relatively simple and a lower frame rate can accurately record the image content), using a fixed frame rate higher than the current lower requirement will result in higher power consumption. Conversely, in situations where the frame rate requirement is high (such as during video calls, where the motion of image content between adjacent frames is more complex and a higher frame rate is needed to record image content more frequently in order to accurately capture motion information), using a fixed frame rate that is too low to meet the higher requirement will result in some motion details between video frames not being captured. Consequently, the video frames will not display these motion details, meaning the display frame rate will also fail to meet the higher requirement, leading to blurry images and reduced display quality. Summary of the Invention

[0006] This application provides a frame rate control method, an electronic device, a storage medium, and a program product.

[0007] In a first aspect, this application provides a frame rate control method, which is applied to a first electronic device. The method includes: a first camera acquiring M consecutive video frames at a first frame rate; the first electronic device determining that the content change degree of the M video frames satisfies the frame rate adjustment condition; and the first camera acquiring N consecutive video frames at a second frame rate. The second frame rate is different from the first frame rate, and the first video frame in the N video frames is acquired after the acquisition of the Mth video frame in the M video frames is completed. M and N are integers greater than 1, and M and N may be the same or different.

[0008] It is understandable that after determining that the content changes of M video frames meet the frame rate adjustment conditions, the first electronic device can determine the acquisition frame rate of N video frames as a second frame rate based on the first frame rate corresponding to the M video frames. Therefore, the first camera can then acquire N video frames at the second frame rate. Thus, in some methods, the first electronic device can determine, based on its internal frame rate determination module, that the content changes of M video frames meet the frame rate adjustment conditions, and then determine that the acquisition frame rate of N video frames is a second frame rate different from the first frame rate.

[0009] Therefore, this method can dynamically adjust the acquisition frame rate of the first camera when acquiring N video frames based on the second frame rate of the N video frames determined by the software-level frame rate determination module according to the content change degree of the M video frames. For example, if the content change degree is large and exceeds the first change degree threshold, the first camera can acquire N video frames at a second frame rate greater than the first frame rate. This approach increases the acquisition frame rate of the first camera when the content change degree of the M video frames is large, which can avoid losing certain motion and change details during the acquisition of video frames.

[0010] For example, if the content changes only slightly and are below a second threshold for change, the first camera captures N video frames at a second frame rate, which is lower than the first frame rate. This method reduces the frame rate of the first camera when the content changes only slightly in M ​​video frames, thereby reducing the power consumption required to capture N video frames, thus mitigating the heat generated by the first electronic device and extending its lifespan.

[0011] In one possible implementation of the first aspect, after the first camera captures N video frames at a second frame rate, the first electronic device encodes and displays the N video frames at the second frame rate.

[0012] Taking the real-time video communication scenario shown in Figure 2 below as an example, after the first electronic device (such as mobile phone 100) determines the second frame rate of N video frames, the first camera can capture the N video frames corresponding to user A at the second frame rate. Then, mobile phone 100 encodes the N video frames at the second frame rate and displays the N video frames on the display screen 101 of mobile phone 100 at the second frame rate for user A to view.

[0013] In this application, N video frames are displayed on the screen of a first electronic device at a second frame rate. Besides a video frame display area capable of displaying the N video frames, the screen may also have other areas, such as a status bar area. For example, as shown in Figure 2, the mobile phone 100 may include a video frame display area 102 and a status bar area 103.

[0014] This application does not limit the display frame rate of the content displayed in the status bar area; it can be the same as or different from the second frame rate. When the display frame rate of the content displayed in the status bar area differs from the second frame rate, the display frame rate can be a frame rate set by the developer of the first electronic device based on experience, or it can be a frame rate selected by the user from multiple frame rate options.

[0015] It is understandable that if the content changes significantly, and the first camera captures N video frames at a second frame rate greater than the first frame rate, then the display frame rate of the N video frames can also be increased. This can avoid the loss of certain motion and change details due to fewer video frames displayed per unit time, which would lead to blurry video frames and reduced display quality.

[0016] It is understandable that if the content changes little and the first camera captures N video frames at a second frame rate lower than the first frame rate, then the display frame rate of the N video frames can also be reduced. This method can reduce the power consumption of capturing the N video frames while ensuring the display quality of the N video frames.

[0017] In one possible implementation of the first aspect, the degree of content variation of the M video frames is determined based on at least one of the following: the first mean square error of the M video frames, the first mean absolute difference of pixels, the first mean texture difference, the first keyframe ratio, and the first quantization parameter difference.

[0018] This application does not limit the method of determining the content change degree of M video frames based on at least one of the first MSE, first MADP, first average texture difference, first I-frame ratio, and first QP difference. For example, a first electronic device can input at least one of the first MSE, first MADP, first average texture difference, first I-frame ratio, and first QP difference of the M video frames into a pre-trained content change degree determination model to obtain the content change degree corresponding to the M video frames.

[0019] In one possible implementation of the first aspect, the first electronic device determines that the content change degree of M video frames satisfies the frame rate adjustment condition, including: the first electronic device determines that the content change degree of M video frames satisfies the frame rate adjustment condition based on a decision tree model, wherein the decision tree model is trained based on a classification and regression tree algorithm.

[0020] It is understood that, in this embodiment of the application, the content change levels of M video frames that satisfy the frame rate adjustment conditions can be determined based on the decision tree model, and a second frame rate corresponding to the content change levels that satisfy the frame rate adjustment conditions can be obtained. The way the decision tree model performs the above process can be seen in the flowchart shown in Figure 6 below, and will not be described in detail here.

[0021] In one possible implementation of the first aspect, the first electronic device determines the acquisition frame rate of N video frames as a second frame rate by: determining that the content change degree of M video frames belongs to a first content change degree range, and obtaining a third frame rate corresponding to the first content change degree range; obtaining a fourth frame rate corresponding to the first network information corresponding to the M video frames, and / or a fifth frame rate corresponding to the first temperature of the first electronic device corresponding to the M video frames; and if the third frame rate is less than the fourth frame rate, and / or the third frame rate is less than the fifth frame rate, the third frame rate is used as the second frame rate.

[0022] It is understandable that this method of determining the second frame rate based on the degree of content change of M video frames, and also based on the first network information of the network and / or the first temperature of the first electronic device, can be seen in the process shown in Figure 10 below, and will not be elaborated here. Compared with the method of determining the acquisition frame rate based solely on network information and device temperature, this application also determines the acquisition frame rate based on the degree of content change, which can make the determination of the acquisition frame rate more accurate.

[0023] In one possible implementation of the first aspect, the method further includes: corresponding to real-time video communication between the first electronic device and the second electronic device, sending N video frames at a second frame rate to the second electronic device, wherein the second electronic device is used to display the N video frames at the second frame rate.

[0024] Taking the real-time video communication scenario shown in Figure 2 below as an example, after the first camera of the first electronic device (such as mobile phone 100) captures N video frames at the second frame rate, the first electronic device can send the N video frames at the second frame rate to mobile phone 200. In this way, mobile phone 200 can display the received N video frames at the second frame rate. This method ensures that the display frame rates of the N video frames of mobile phone 100 and mobile phone 200 are adjusted synchronously, improving the synchronization of the content displayed by mobile phone 100 and mobile phone 200.

[0025] In a second aspect, this application provides an electronic device comprising: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the frame rate control method of the first aspect and any possible implementation thereof.

[0026] Thirdly, this application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the frame rate control method of the first aspect and any possible implementation thereof.

[0027] Fourthly, this application provides a computer program product comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform the frame rate control method of the first aspect and any possible implementation thereof.

[0028] The beneficial effects of the second to fourth aspects can be found in the first aspect and the beneficial effects of any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0029] Figure 1 illustrates a schematic diagram of the relationship between the user's experience quality of interface content and the display frame rate when making a video call using an electronic device, according to some embodiments of this application.

[0030] Figure 2 illustrates a video call scenario according to some embodiments of this application;

[0031] Figure 3A shows a timing diagram corresponding to a first type of M video frames and N video frames according to some embodiments of this application;

[0032] Figure 3B illustrates a timing diagram corresponding to the second type of M video frames and N video frames according to some embodiments of this application;

[0033] Figure 4 illustrates a timing diagram of the third type of M video frames and N video frames according to some embodiments of this application;

[0034] Figure 5 shows a flowchart of a frame rate control method according to some embodiments of this application;

[0035] Figure 6 illustrates a decision logic diagram of a decision tree model according to some embodiments of this application;

[0036] Figure 7 illustrates a system architecture diagram of a first electronic device according to some embodiments of this application;

[0037] Figure 8 illustrates an interactive diagram of various modules within a first electronic device according to some embodiments of this application;

[0038] Figure 9 illustrates an interactive flow diagram of a frame rate control method in a video call scenario, according to some embodiments of this application.

[0039] Figure 10 illustrates a flowchart of determining the acquisition frame rate based on the degree of content change, first network information, and first temperature, according to some embodiments of this application.

[0040] Figure 11 illustrates a flowchart of a method for determining a second frame rate according to some embodiments of this application;

[0041] Figure 12 shows a schematic diagram of the hardware structure of an electronic device according to some embodiments of this application. Detailed Implementation

[0042] The illustrative embodiments of this application include, but are not limited to, frame rate control methods, electronic devices, storage media, and program products.

[0043] The following is a brief introduction to the proper nouns used in this application.

[0044] Inter-frame motion complexity: The complexity of motion and change between adjacent video frames can be represented by the difference in pixel values ​​between adjacent video frames. For example, the greater the difference in pixel values ​​between adjacent video frames, the higher the inter-frame motion complexity.

[0045] Mean squared error (MSE): In this application, it is used to represent the difference between adjacent video frames, and is obtained by calculating the average of the squared differences of each pixel. The MSE of adjacent video frames can reflect the degree of content change between adjacent video frames. For example, the larger the MSE, the greater the degree of content change and the higher the inter-frame motion complexity of adjacent video frames.

[0046] Mean absolute difference per pixel (MADP): This represents the difference between adjacent video frames. It is obtained by calculating the average of the absolute differences of each pixel. The MADP of adjacent video frames can reflect the degree of content change between them. For example, the larger the MADP, the greater the degree of content change and the higher the inter-frame motion complexity between adjacent video frames.

[0047] Texture complexity: This describes the richness of texture features in a video frame and can be represented by the mean absolute deviation of the image (MADI). MADI is calculated by taking the mean of the absolute differences between each pixel value in a video frame and the average value. Therefore, the difference in MADI between adjacent video frames represents the texture difference. It can be understood that the greater the texture difference between adjacent video frames, the greater the degree of content variation, and the higher the inter-frame motion complexity.

[0048] I-frame ratio: The proportion of I-frames out of the total number of video frames in a given set of video frames. An I-frame is a keyframe in video coding, containing complete image data and requiring no other frames for decoding. The I-frame ratio reflects the degree of content variation between adjacent video frames; for example, a higher I-frame ratio indicates greater content variation and higher inter-frame motion complexity between adjacent video frames.

[0049] Quantization parameter (QP): In video coding, QP controls the quality and compression ratio of video compression, reflecting the compression intensity. The QP difference between adjacent video frames can indirectly reflect the degree of content change. For example, the larger the QP difference, the greater the degree of content change and the higher the inter-frame motion complexity of adjacent video frames.

[0050] This application does not limit the application scenarios of the frame rate control method; for example, it can include real-time video communication scenarios. Real-time video communication scenarios can include video call scenarios, live streaming scenarios, online conferencing scenarios, etc. The following description uses the application of the frame rate control method to a real-time video communication scenario, specifically a video call scenario, as an example.

[0051] Figure 2 illustrates a video call scenario. Taking a mobile phone as an example, as shown in Figure 2, user A uses mobile phone 100 to conduct a video call with user B using mobile phone 200. During the video call, the first camera of mobile phone 100 captures video frames corresponding to user A at a certain frame rate. Then, mobile phone 100 displays the captured video frames corresponding to user A at that frame rate in the video frame display area 102 of display interface 101 for user A to view. In addition, mobile phone 100 can also send the video frames corresponding to user A at that frame rate to mobile phone 200 via the network. Thus, the display interface 201 of mobile phone 200 can also display the video frames corresponding to user A at that frame rate for user B to view. Based on the same principle, the display interfaces 201 of mobile phone 200 and 100 also display video frames corresponding to user B captured by the camera (e.g., the second camera) of mobile phone 200.

[0052] In addition, the display interface 101 of the mobile phone 100 also includes a status bar area 103. It can be understood that the display frame rate of the video frame corresponding to user A in the video frame display area 102 and the display frame rate of the content displayed in the status bar area 103 may be the same or different, and this application does not limit this.

[0053] As mentioned above, in some methods, the camera uses a fixed frame rate when capturing video frames, which in turn causes the electronic device to use a fixed display frame rate when displaying the captured video frames. However, if the fixed capture frame rate is higher than the required capture frame rate, it will result in high power consumption during video frame capture. If the fixed capture frame rate is lower than the required capture frame rate, the camera will be unable to accurately capture motion information, resulting in blurry video frames and low display quality.

[0054] Therefore, to solve the above-mentioned technical problems, this application provides a frame rate control method. In this method, a first electronic device can determine the required acquisition frame rate based on the content of M consecutive video frames in the previous time interval. Then, the first camera of the first electronic device acquires N consecutive video frames in the next time interval at the determined acquisition frame rate. Specifically, the first camera can acquire M consecutive video frames in the previous time interval at a first frame rate. Then, the first electronic device determines the degree of content change of the M video frames and obtains a second frame rate corresponding to the N video frames based on the degree of content change. Next, the first camera acquires N consecutive video frames in the next time interval of the previous time interval at the second frame rate. For example, if the degree of content change is large (e.g., greater than a first change threshold), the first camera can acquire N consecutive video frames at a second frame rate greater than the first frame rate; if the degree of change is small (e.g., less than the second change threshold), the first camera can acquire N consecutive video frames at a second frame rate less than the first frame rate. It is understandable that the degree of content change can represent the complexity of motion and change between video frames. Therefore, the higher the degree of content change, the more significant the motion and change between video frames. In this case, a higher frame rate is required to avoid losing certain motion details during video frame acquisition.

[0055] In this embodiment, the first video frame of N video frames is acquired after the Mth video frame of M video frames has been acquired. M and N are both integers greater than 1. The values ​​of M and N can be the same or different. This application does not limit this.

[0056] In some embodiments, when the content variation of M video frames falls within a first content variation range, the first electronic device may use the frame rate corresponding to the first content variation range as the second frame rate. The correspondence between the first content variation range and the frame rate may be preset based on experience.

[0057] In some embodiments, after the first camera captures N video frames at a second frame rate, the first electronic device can display the captured N video frames on the display screen of the first electronic device at the second frame rate.

[0058] Figure 3A illustrates a timing diagram of M and N video frames during display. Referring to Figure 3A, time T0 represents the start time of displaying the M video frames, time T1 represents the end time of displaying the M video frames (or the start time of displaying the N video frames), and time T2 represents the end time of displaying the N video frames. The interval between time T0 and time T1 can be considered a first time interval, and the interval between time T1 and time T2 can be considered a second time interval. In the first time interval, the display frame rate of the M video frames is the same as the acquisition frame rate, which is the first frame rate F1. After the first electronic device determines the second frame rate F2 based on the degree of content change in the acquired M video frames, the first camera acquires N consecutive video frames at the second frame rate F2. Subsequently, the first electronic device displays the N consecutive video frames at the second frame rate within the second time interval.

[0059] It is understandable that the first electronic device can determine the degree of content change of the M video frames after acquiring the M video frames captured by the first camera, and then determine the acquisition frame rate of the N video frames. Referring to Figure 3A, if the N video frames are acquired before the end time of the display of the M video frames (i.e., time T1), then the start time of the display of the N video frames is the same as the end time of the display of the M video frames, that is, time T1.

[0060] In this method, after M video frames are displayed at the first frame rate, N video frames can begin to be displayed at the second frame rate. Since the second frame rate is determined based on the degree of content change, the display quality of the N video frames can be improved.

[0061] In Figure 3A above, the end point of the first time interval (time T1) and the start point of the second time interval are the same. In some other embodiments, the end point of the first time interval and the start point of the second time interval may be different. For example, as shown in Figure 3B, time T0 represents the start time of displaying M video frames, time T1 represents the end time of displaying M video frames, time T2 represents the end time of displaying N video frames, and time T3 represents the start time of displaying N video frames. The interval between time T0 and time T1 can be the first time interval, and the interval between time T3 and time T2 can be the second time interval. It can be understood that if N video frames are captured after the end time of displaying M video frames (i.e., time T1), for example, before time T3, then the start time of displaying N video frames (time T3) is different from the end time of displaying M video frames (time T1), meaning there is a certain time interval between the start point of the second time interval (time T3) and the end point of the first time interval (time T1).

[0062] It is understood that regardless of whether the end point of the first time interval and the start point of the second time interval are the same or different, the first video frame of the N consecutive video frames in the second time interval can be displayed after the Mth video frame of the M consecutive video frames in the first time interval has been displayed. Here, M and N are both integers greater than 1, and the values ​​of M and N can be the same or different; this embodiment does not limit this.

[0063] The following example illustrates the sequential relationship between M and N video frames during display, assuming that the end point of the first time interval and the start point of the second time interval are the same.

[0064] Referring to Figure 4, the interval between time T0 and time T1 can be considered the first time interval, and the interval between time T1 and time T2 can be considered the second time interval. As shown in Figure 4, the end point (time T1) of the first time interval and the start point (time T2) of the second time interval are the same. Furthermore, the first time interval displays M consecutive video frames, and the second time interval displays N consecutive video frames. As shown in Figure 4, the first video frame out of the N video frames is displayed after the Mth video frame out of the M video frames has been displayed.

[0065] In some embodiments, the degree of content change of the M video frames is the inter-frame motion complexity of the M video frames, and the inter-frame motion complexity can be obtained based on at least one of the first MSE, first MADP, first I-frame ratio, first texture difference mean, and first QP difference of the M video frames.

[0066] It is understood that the above content only relates to the method of determining the second frame rate based on the degree of content change. In some other embodiments, the second frame rate can also be determined based on the first network information of the network where the M video frames are located, and / or the temperature of the first electronic device. In this case, the method of determining the second frame rate may include: when the degree of content change of the M video frames is within a first range of content change, using the frame rate corresponding to the first range of content change as the third frame rate; obtaining the fourth frame rate corresponding to the first network information, and / or obtaining the fifth frame rate corresponding to the first temperature of the first electronic device. Then, when the third frame rate is less than the fourth frame rate and / or the fifth frame rate, the third frame rate is used as the second frame rate, and N video frames are acquired and displayed at the second frame rate. This method can select the smallest frame rate among the third frame rate determined based on the degree of content change, the fourth frame rate determined based on the first network information, and the fifth frame rate determined based on the temperature of the first electronic device, and acquire N video frames at this frame rate, which can reduce the power consumption of acquiring N video frames.

[0067] The first electronic device inputs first network information (such as transmission bitrate, reception bitrate, latency, jitter, packet loss, etc.) within a time interval corresponding to M consecutive video frames into a pre-trained transmission bitrate determination model to obtain the predicted transmission bitrate within the time interval corresponding to N video frames. Then, based on the predicted transmission bitrate, the first electronic device looks up the code table to obtain the fourth frame rate corresponding to the transmission bitrate. The code table contains multiple correspondences between transmission bitrates and multiple frame rates.

[0068] It is understood that the training set corresponding to the pre-trained transmit bitrate determination model includes network information for at least one time interval used for training (e.g., transmit bitrate, receive bitrate, latency, jitter, packet loss, etc.), and a reference transmit bitrate for the next time interval of each time interval. During training, the network information used for training is input into the model to obtain the predicted transmit bitrate for the next time interval corresponding to each time interval. Then, the model is repeatedly trained based on the difference between the predicted transmit bitrate and the reference transmit bitrate until a trained transmit bitrate determination model is obtained. Next, the first electronic device can input the first network information within the time intervals corresponding to the above M video frames into the trained transmit bitrate determination model to obtain the transmit bitrate within the time intervals corresponding to the N video frames output by the model.

[0069] Furthermore, the first electronic device can also input its first temperature within the time interval corresponding to M video frames into a pre-trained device temperature determination model to obtain a predicted second temperature for the first electronic device within the time interval corresponding to N video frames. Then, the first electronic device can look up a correspondence table between multiple frame rates and multiple temperatures based on the predicted second temperature to obtain a fifth frame rate corresponding to the predicted second temperature.

[0070] It can be understood that the training set corresponding to the pre-trained device temperature determination model includes the device temperature corresponding to at least one time interval used for training, and the reference temperature of the device corresponding to the next time interval of each time interval. During training, the device temperature corresponding to at least one time interval used for training is input into the model to obtain the predicted temperature of the device for the next time interval corresponding to each time interval. Then, the model is repeatedly trained based on the difference between the predicted temperature and the reference temperature until the trained device temperature determination model is obtained. Next, the first electronic device can input its first temperature within the time interval corresponding to the above M video frames into the trained device temperature determination model to obtain the second temperature of the first electronic device within the time interval corresponding to the N video frames output by the model.

[0071] In some embodiments, in a scenario where the first electronic device and the second electronic device are engaged in real-time video communication, after the first camera captures N video frames at a second frame rate, the first electronic device can also send the N video frames at the second frame rate to the second electronic device. Thus, the second electronic device can also display the N video frames sent by the first electronic device at the second frame rate.

[0072] In the method provided in this application, the second frame rate corresponding to N video frames is determined based on the degree of content change corresponding to M video frames. In other words, the acquisition frame rate when the first camera acquires N video frames and the display frame rate when the first electronic device displays N video frames can change in real time based on the degree of content change. This method can improve the display quality of N video frames or reduce the acquisition power consumption of N video frames.

[0073] Before detailing the frame rate control method of this application, the electronic devices to which the method provided in the embodiments of this application can be applied are first described. It is understood that the method provided in this application can be applied to any electronic device capable of capturing and displaying the captured video frames on a screen, including but not limited to mobile stations (MS) and mobile terminals (MT). For example, the electronic device can be a mobile phone, smart TV, wearable device, tablet computer, desktop computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, terminal in industrial control, terminal in self-driving, terminal in remote medical surgery, terminal in smart grid, terminal in transportation safety, terminal in smart city, terminal in smart home, etc. The embodiments of this application do not limit the specific form of the electronic device.

[0074] The frame rate control method provided in this application will now be described in detail with reference to the accompanying drawings. Figure 5 shows a flowchart of a frame rate control method provided in this application, which can be applied to a first electronic device including a first camera. As shown in Figure 5, the method includes, but is not limited to, the following schemes:

[0075] 501: The first camera captures M consecutive video frames at the first frame rate.

[0076] This application does not limit the scenario where the first camera captures M consecutive video frames, such as a real-time video communication scenario. Specifically, in the real-time video communication scenario of a video call as shown in Figure 2, taking a mobile phone 100 as the first electronic device, the first camera of the mobile phone 100 can capture M video frames corresponding to user A during a video call between user A and user B.

[0077] It is understood that after the first camera captures M consecutive video frames at a first frame rate, the first electronic device can display the M video frames on the display screen at the first frame rate (as shown in Figure 2, for example). Exemplarily, the first electronic device can display the M video frames on the display screen at the first frame rate based on the display module. However, this application does not limit the method for determining the first frame rate; for example, the first frame rate can be a frame rate preset by the developer of the first electronic device based on experience.

[0078] 502: The first electronic device determines whether the content changes of the M video frames meet the frame rate adjustment conditions.

[0079] In some embodiments, after acquiring M video frames, the first electronic device can further determine the degree of content change in the M video frames. For example, the first electronic device can determine the degree of content change in the M video frames based on an internal content change degree determination module.

[0080] This application does not limit the timing of when the first electronic device determines the degree of content change of the M video frames. For example, the first electronic device can determine the content information of the M video frames after acquiring them, such as during the encoding of the M video frames, and then determine the degree of content change based on the content information. For example, during the encoding of the M video frames, the encoder can determine the content information of the M video frames (e.g., MSE, MADP, I-frame ratio, texture complexity, QP value), and then determine the degree of content change corresponding to the M video frames based on one or more of the aforementioned content information.

[0081] In this embodiment, the degree of content change of the M video frames can be understood as the complexity of the motion and changes of the content between video frames (i.e., inter-frame motion complexity), representing the degree of difference between video frames. Therefore, it can be understood that if two adjacent video frames are completely identical, that is, there is no difference between the two video frames, then the degree of content change is zero. The specific method for determining the degree of content change of the M video frames will be described in detail later, and will not be repeated here.

[0082] After the first electronic device determines the degree of content change in M ​​video frames, it can further determine whether the degree of content change meets the frame rate adjustment conditions. If the degree of content change does not meet the frame rate adjustment conditions, then step 503 is executed, and the first camera continues to capture N consecutive video frames at the first frame rate. If the degree of content change meets the frame rate adjustment conditions, then step 504 is executed, and the first camera captures N consecutive video frames at a second frame rate different from the first frame rate. It can be understood that in this case, the second frame rate can be greater than or less than the first frame rate.

[0083] In some embodiments, the degree of content change can be compared with a preset degree of change threshold to determine whether the degree of content change of M video frames meets the frame rate adjustment condition. For example, the frame rate adjustment condition can be that the degree of content change is greater than a first degree of change threshold, or that the degree of content change is less than a second degree of change threshold. The first degree of change threshold and the second degree of change threshold can be the same or different, and this embodiment does not limit this.

[0084] Furthermore, the embodiments of this application do not limit the method of determining the first degree of change threshold and the second degree of change threshold. For example, the first degree of change threshold and the second degree of change threshold can be set based on experience or flexibly adjusted based on the actual application scenario.

[0085] It is understood that the above frame rate adjustment conditions are merely illustrative examples and do not constitute a complete limitation on this application. In other words, the frame rate adjustment conditions can be any conditions that determine whether the acquisition frame rate needs to be adjusted based on the degree of content change of M video frames.

[0086] It is understandable that after the first electronic device determines that the degree of content change meets the frame rate adjustment conditions, it can determine the second frame rate corresponding to N video frames based on the first frame rate. Then, the first camera can capture N video frames at the second frame rate. Therefore, in some embodiments, the first electronic device can determine whether the degree of content change of M video frames meets the frame rate adjustment conditions based on its internal frame rate determination module, and if the frame rate adjustment conditions are met, further determine that the capture frame rate of N video frames is the second frame rate.

[0087] 503: The first camera captures N consecutive video frames at the first frame rate.

[0088] The first video frame out of N video frames is acquired after the Mth video frame out of M video frames has been acquired. M and N are integers greater than 1, and M and N may be the same or different.

[0089] For example, taking the frame rate adjustment condition as the content change degree being greater than the first change degree threshold, or the content change degree being less than the second change degree threshold, it can be understood that if the content change degree is greater than or equal to the second change degree and less than or equal to the first change degree threshold, that is, the content change degree does not meet the frame rate adjustment condition, then the first camera will still collect N consecutive video frames at the first frame rate.

[0090] 504: The first camera captures N consecutive video frames at the second frame rate.

[0091] The second frame rate is different from the first frame rate, and the first video frame out of N video frames is acquired after the Mth video frame out of M video frames has been acquired.

[0092] In the embodiments of this application, M and N are integers greater than 1, and M and N can be the same or different, which is not limited in this embodiment.

[0093] Taking a frame rate adjustment condition where the content change level is greater than a first change level threshold, or the content change level is less than a second change level threshold, as an example, if the content change level is greater than the first change level threshold, the first electronic device can increase the first frame rate. In this way, the first camera can capture N video frames at a second frame rate greater than the first frame rate. For example, if the content change level is greater than the first change level threshold and falls within the first content change level range, the first electronic device can use the frame rate corresponding to the first content change range as the second frame rate. It is understood that this application embodiment does not limit the correspondence between the first content change range and the frame rate; this correspondence can be set based on experience.

[0094] When the content of the M video frames changes significantly, this method increases the frame rate of the first camera for capturing the N video frames, which can prevent the loss of certain motion and change details during the acquisition process.

[0095] Furthermore, if the degree of content change is less than a second change threshold, the first electronic device can reduce the first frame rate. In this way, the first camera can capture N video frames at a second frame rate lower than the first frame rate. For example, if the degree of content change is less than the second change threshold and falls within the second content change range, the first electronic device will use the frame rate corresponding to the second content change range as the second frame rate. This method, when the content change of M video frames is relatively small, reduces the frame rate at which the first camera captures N video frames, thereby reducing the power consumption required to capture N video frames, thus mitigating the heat generated by the first electronic device and extending its lifespan.

[0096] In some other embodiments, after the first camera captures N video frames at a second frame rate, the first electronic device can also encode the N video frames at the second frame rate and display the N video frames on the display screen.

[0097] Taking the real-time video communication scenario shown in Figure 2 as an example, after the first electronic device (such as mobile phone 100) determines the second frame rate of N video frames, the first camera can capture the N video frames corresponding to user A at the second frame rate. Then, mobile phone 100 encodes the N video frames at the second frame rate and displays the N video frames on the display screen 101 of mobile phone 100 at the second frame rate for user A to view.

[0098] It is understandable that if the content changes significantly, the first electronic device can display N video frames at a second frame rate greater than the first frame rate. This method can avoid the loss of certain motion and change details due to fewer video frames displayed per unit time, which would lead to blurry video frames and reduced display quality.

[0099] If the content changes only slightly, the first electronic device can display N video frames at a second frame rate lower than the first frame rate. This method can ensure the display quality of N video frames while reducing the power consumption of acquiring N video frames.

[0100] In this application, the display screen may have other areas besides a display area capable of displaying N video frames, such as a status bar area. Taking Figure 2 as an example, the mobile phone 100 may include a video frame display area 102 and a status bar area 103.

[0101] This application does not limit the display frame rate of the content corresponding to the status bar area. It can be the same as or different from the second frame rate corresponding to N video frames. When the display frame rate of the content corresponding to the status bar area is different from the second frame rate, the display frame rate corresponding to the status bar area can be the frame rate set by the developer of the first electronic device based on experience, or it can be the frame rate selected by the user from multiple frame rate options.

[0102] The following describes in detail how the degree of content change in M ​​video frames is determined.

[0103] In this embodiment of the application, the degree of content change of M video frames can be determined based on content information, and the content information includes at least one of the following: first MSE, first MADP, first texture difference mean, first I-frame ratio, and first QP difference.

[0104] In some embodiments, the Mean Squared Es (MSE) of adjacent video frames in M ​​video frames can be obtained by calculating the average of the squared differences of each pixel. Therefore, the first electronic device can again average the multiple MSEs corresponding to multiple adjacent video frames in the M video frames to obtain the first MSE corresponding to the M video frames. It can be understood that the first MSE is positively correlated with the degree of content change; that is, the larger the first MSE, the greater the difference between pixels, and the greater the degree of content change.

[0105] In some embodiments, the MADP of adjacent video frames in M ​​video frames can be obtained by calculating the average of the absolute differences of each pixel. Therefore, the first electronic device can again average the multiple MADPs corresponding to multiple adjacent video frames in the M video frames to obtain the first MADP corresponding to the M video frames. It can be understood that the first MADP is positively correlated with the degree of content change; that is, the larger the first MADP, the greater the difference between pixels, and the greater the degree of content change.

[0106] In some embodiments, the texture complexity of any video frame among the M video frames can be represented by MADI (Mean Absolute Distortion), where MADI is obtained by calculating the average of the absolute differences between each pixel value in the video frame and the average value. Therefore, the texture difference between adjacent video frames among the M video frames can be obtained based on the difference in MADI between adjacent video frames. Further, the first electronic device can again average the differences in MADI corresponding to multiple adjacent video frames among the M video frames to obtain the first texture difference mean value corresponding to the M video frames. It can be understood that the first texture difference mean value is positively correlated with the degree of content change; that is, the larger the first texture difference mean value, the greater the degree of content change.

[0107] In some embodiments, the first I-frame ratio represents the proportion of I-frames out of the total M video frames. It can be understood that an I-frame contains complete frame data and can be decoded independently. When there are significant changes in content between different video frames, more I-frames are needed to capture these changes to ensure that each video frame can be clearly represented. Therefore, a higher first I-frame ratio indicates a greater degree of content variation.

[0108] In some embodiments, each of the M video frames corresponds to a QP value during encoding. Therefore, the difference between the QP values ​​of adjacent video frames in the M video frames is the QP difference between adjacent video frames. Further, the first electronic device can again average the multiple QP differences corresponding to multiple adjacent video frames in the M video frames to obtain the first QP difference corresponding to the M video frames. It can be understood that if the first QP difference between adjacent video frames is large, it indicates a large difference in compression intensity between the adjacent video frames, and thus a large degree of content variation between the adjacent video frames.

[0109] This application does not limit the method of determining the content change degree of M video frames based on at least one of the first MSE, first MADP, first average texture difference, first I-frame ratio, and first QP difference. For example, a first electronic device can input at least one of the first MSE, first MADP, first average texture difference, first I-frame ratio, and first QP difference of the M video frames into a pre-trained content change degree determination model to obtain the content change degree corresponding to the M video frames.

[0110] It is understood that the training set for the pre-trained content change determination model includes at least one of the following: MSE, MADP, mean texture difference, I-frame ratio, and QP difference, as well as the reference content change degree. During training, at least one of the following training parameters—MSE, MADP, mean texture difference, I-frame ratio, and QP difference—is input into the model to obtain the predicted content change degree. Then, the model is repeatedly trained based on the difference between the predicted content change degree and the reference content change degree until the trained content change determination model is obtained.

[0111] In other embodiments, the first electronic device may determine the second frame rate of N video frames based on a pre-trained frame rate determination model. This application does not limit the type of frame rate determination model; for example, the frame rate determination model may be a neural network model or a machine learning model (e.g., a decision tree model).

[0112] The following section describes in detail the training and usage process of the model, taking the frame rate determination model as a decision tree model and using the model to reduce the first frame rate to obtain the second frame rate.

[0113] During training, the training set for this decision tree model can include content information for multiple videos displayed at a first training frame rate, as well as labels for each video. Each video can include multiple consecutive video frames, and the label for each video includes a pre-labeled reference frame rate, such as one manually labeled by the user. For example, for a given video, the user can score its display quality at different frame rates. Taking a video with two frame rates (e.g., 15FPS and 10FPS) as an example, the user can score the display quality of the video displayed at 15FPS (the first training frame rate), obtaining a first score. The user can also score the display quality of the video displayed at 10FPS, obtaining a second score. If the difference between the first score and the second score is less than a certain score threshold (e.g., 0.5), it indicates that the difference in display quality between the video displayed at 15FPS and 10FPS is small, and the video can be downgraded to be displayed at 10FPS. In this case, the reference frame rate for the video is 10 FPS.

[0114] It is understood that the first electronic device can extract features from multiple videos displayed at the first training frame rate to obtain content information. This content information may include, but is not limited to, MSE, MADP, mean texture difference, I-frame ratio, and QP difference.

[0115] Then, the first electronic device inputs the aforementioned information into the decision tree model to obtain the predicted frame rate output by the decision tree model. It is understood that in this embodiment, the decision tree model can employ the classification and regression trees (CART) algorithm, using Gini index for feature selection to obtain the predicted frame rate. Next, the first electronic device repeatedly trains the model based on the difference between the predicted frame rate and the reference frame rate until a fully trained decision tree model is obtained. It is understood that, since the frame rate at which the first camera captures N videos and the display frame rate at which the display module of the first electronic device displays N video frames are the same in this application, the display frame rate of the N video frames obtained based on the trained decision tree model can also be used as the frame rate at which the first camera captures N video frames.

[0116] In some embodiments, the first electronic device can input the content information (including but not limited to the first MSE, first MADP, first I-frame ratio, and first QP difference) corresponding to the M video frames in this application into a trained decision tree model to obtain the second frame rate corresponding to the M video frames. Then, the first camera acquires N video frames at the second frame rate.

[0117] Taking a first frame rate of 15 FPS and a second frame rate of 10 FPS as an example, Figure 6 shows a schematic diagram of the decision logic of a trained decision tree model. As shown in Figure 6, the decision logic of this model includes, but is not limited to, the following schemes:

[0118] 600: Get M video frames.

[0119] It is understandable that the first electronic device can acquire M video frames captured by the first camera at the first frame rate.

[0120] 601: Determine whether the first MSE of M video frames is less than or equal to the first threshold.

[0121] Understandably, if the first MSE (Std_aveMse) of the M video frames is less than or equal to the first threshold, then execute step 602 to determine that the display frame rate of the N video frames is a second frame rate that is less than the first frame rate. If not, that is, the first MSE of the M video frames is greater than the first threshold, then execute step 603.

[0122] In some embodiments, the first threshold can be 4.195. It can be understood that if the first MSE of the M video frames is less than or equal to the first threshold, it means that the first MSE of the M video frames is small, the difference between pixels is small, that is, the content change is small. At this time, the first frame rate can be reduced to obtain the second frame rate.

[0123] 602: Determine that the acquisition frame rate of N video frames is the second frame rate, which is less than the first frame rate.

[0124] 603: Determine whether the first MADP of M video frames is less than or equal to the second threshold.

[0125] Understandably, if the first MADP (Avg_madP) of the M video frames is less than or equal to the second threshold, then execute step 604 to further determine the texture complexity (Avg_madISum) of the M video frames. If not, that is, the MADP of the M video frames is greater than the second threshold, then execute step 605 to further determine the maximum image quality target (Max_IRatio) of the M video frames.

[0126] In some embodiments, the second threshold may be 98.096.

[0127] 604: Determine whether the texture complexity of M video frames is less than or equal to the third threshold.

[0128] Understandably, if the texture complexity (Avg_madISum) of the M video frames is less than or equal to the third threshold, then execute 606 to further determine the maximum image quality target (Max_IRatio) of the M video frames. If not, that is, if the texture complexity of the M video frames is greater than the third threshold, then execute 602 to reduce the first frame rate and display the N video frames at a second frame rate that is lower than the first frame rate.

[0129] In some embodiments, the third threshold may be 99.466.

[0130] 605: Determine whether the maximum image quality target of M video frames is less than or equal to the fourth threshold.

[0131] Understandably, if the maximum image quality target (Max_IRatio) of the M video frames is less than or equal to the fourth threshold, then execute step 607 to further determine the first QP difference (Avg_diff_aveQp) of the M video frames. If not, that is, the average texture difference of the M video frames is greater than the fourth threshold, then execute step 602.

[0132] It is understandable that the maximum image quality target is positively correlated with the I-frame ratio; that is, the larger the I-frame ratio, the higher the maximum image quality target. Therefore, the maximum image quality target can reflect the size of the I-frame ratio.

[0133] In some embodiments, the fourth threshold may be 91.

[0134] 606: Determine whether the maximum image quality target of M video frames is less than or equal to the fifth threshold.

[0135] It is understandable that if the maximum image quality target (Max_IRatio) of the M video frames is less than or equal to the fifth threshold, then execute step 602. If not, that is, if the maximum image quality target of the M video frames is greater than the fifth threshold, then execute step 608 to further determine whether the maximum image quality target of the M video frames is less than or equal to the seventh threshold.

[0136] In some embodiments, the fifth threshold may be 3.712.

[0137] 607: Determine whether the difference in the first QP of M video frames is less than or equal to the sixth threshold.

[0138] Understandably, if the first QP difference (Avg_diff_aveQp) of the M video frames is less than or equal to the sixth threshold, then execute step 609 to further determine whether the first MSE of the M video frames is less than or equal to the eighth threshold. If not, that is, the first QP difference of the M video frames is greater than the sixth threshold, then execute step 602.

[0139] In some embodiments, the sixth threshold can be 0.381.

[0140] 608: Determine whether the maximum image quality target of M video frames is less than or equal to the seventh threshold.

[0141] It is understandable that if the maximum image quality target of the M video frames is less than or equal to the seventh threshold, then step 602 is executed. If not, then step 610 is executed, without adjusting the frame rate, and the display frame rate of the N video frames is determined to be the first frame rate.

[0142] In some embodiments, the seventh threshold can be 36.5.

[0143] 609: Determine whether the first MSE of M video frames is less than or equal to the eighth threshold.

[0144] It is understandable that if the first MSE of the M video frames is less than or equal to the eighth threshold, then step 602 is executed. If not, then step 610 is executed, without adjusting the frame rate, to determine the display frame rate of the N video frames as the first frame rate.

[0145] In some embodiments, the eighth threshold can be 43.855.

[0146] 610: Determine the frame rate of the N video frames to be the first frame rate.

[0147] It is understood that the above description only uses the example of reducing the first frame rate to obtain a second frame rate lower than the first frame rate to illustrate the training and usage of the decision tree model. However, it should be understood that it is also possible to increase the first frame rate based on the decision tree model to obtain a second frame rate higher than the first frame rate. The training and usage of the decision tree model in this case is similar to that when reducing the first frame rate, and will not be elaborated further here.

[0148] Figure 7 illustrates a system architecture diagram of a first electronic device. As shown in Figure 7, the operating system of the first electronic device can be divided into four layers: the application layer, the application framework layer, the system runtime library layer, and the kernel layer.

[0149] The application layer package may include applications such as video communication, camera, Bluetooth, music, SMS, gallery, and map. In this embodiment, when video frames need to be captured, the video communication application in the application layer can initiate a video frame capture request, wherein the capture request may include the required capture frame rate (such as a second frame rate). In some embodiments, the video communication application (program) can determine a second frame rate for N video frames based on the content variation of M video frames, in addition to the first frame rate.

[0150] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. As shown in Figure 7, the application framework layer may include camera services, window managers, content managers, resource managers, notification managers, and phone managers, etc. For example, the application framework layer can provide APIs related to camera services for video communication applications, enabling the video communication applications to call the first camera based on the provided APIs to capture video frames.

[0151] The system runtime layer connects the kernel layer and the application framework layer, providing a series of functional interfaces that enable applications to communicate with the actual hardware, including interfaces related to camera calls. In addition, as shown in Figure 7, the system runtime layer can also include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), and a 2D graphics engine. For example, for video frames captured by the first camera, the 2D graphics engine in the system runtime layer can perform encoding operations such as format conversion on the captured video frames, enabling the application to accurately load and display the video frames.

[0152] The kernel layer is responsible for connecting the system runtime library layer and hardware modules. As shown in Figure 7, the kernel layer includes camera drivers, display drivers, audio drivers, sensor drivers, etc. Among them, the camera driver is responsible for interacting with the system runtime library layer and the camera hardware (such as the first camera). For example, the camera driver can receive a second frame rate from the system runtime library layer and control the first camera to capture video frames at that second frame rate. It can be understood that the video frames captured at the second frame rate can be obtained and displayed by the application layer based on the kernel layer, the system runtime library layer, and the application framework layer.

[0153] Based on Figure 7, Figure 8 shows an interactive schematic diagram of the various modules inside a first electronic device. As shown in Figure 8, the first electronic device may include a content change degree determination module, a frame rate determination module, a display module, and a camera driver module. Based on this, the interaction method includes:

[0154] 801: The content change degree determination module acquires M consecutive video frames and determines the content change degree of the M video frames.

[0155] Among them, M video frames are acquired by the first camera at the first frame rate. Therefore, the content change degree determination module in the first electronic device can obtain M video frames from the first camera. In addition, the content change degree determination module can be located in the system runtime library layer shown in Figure 7.

[0156] The method for determining the degree of content change in the M video frames has been explained above and will not be repeated here.

[0157] 802: The content change level determination module sends the content change level to the frame rate determination module.

[0158] The frame rate determination module can be located in the video communication application within the application layer of Figure 7.

[0159] 803: The frame rate determination module adjusts the first frame rate based on the degree of content change to obtain the second frame rate.

[0160] As can be understood, based on the preceding text, if the frame rate determination module determines that the degree of content change meets the frame rate adjustment conditions, a second frame rate can be obtained based on the first frame rate. The method for determining the second frame rate has been detailed above and will not be repeated here.

[0161] 804: The frame rate determination module sends the second frame rate to the camera driver module.

[0162] The camera driver module can be located in the kernel layer shown in Figure 7.

[0163] 805: The camera driver module drives the first camera to capture N video frames at the second frame rate.

[0164] It is understandable that after the camera driver module obtains the second frame rate, it can drive the first camera to capture video frames at the second frame rate and obtain N video frames captured by the first camera.

[0165] 806: The camera driver module sends N video frames at the second frame rate to the display module.

[0166] The display module can be located in the video communication application within the application layer of Figure 7.

[0167] 807: The display module displays N video frames on the screen at the second frame rate.

[0168] The display module can display N video frames on the display screen at a second frame rate based on the display driver in Figure 7.

[0169] It is understood that the method provided in this application determines the capture frame rate when the first camera in the first electronic device captures N video frames, and the display frame rate when the first electronic device displays N video frames, based on the degree of content change in M ​​video frames. In other words, the method of this application can adjust the capture frame rate of the first camera capturing N video frames and the display frame rate of the N video frames displayed on the screen of the first electronic device in real time based on the changes in the content change of the M video frames. This approach can increase the capture frame rate of the first camera on the N video frames when the content change of the M video frames is high, thereby increasing the display frame rate of the N video frames by the first electronic device, which can improve the display quality of the N video frames. Alternatively, this approach can decrease the capture frame rate of the first camera on the N video frames when the content change of the M video frames is low, thereby decreasing the display frame rate of the N video frames by the first electronic device. This can reduce the power consumption of capturing the N video frames while maintaining display quality, thus increasing the lifespan of the first electronic device.

[0170] Furthermore, this method adjusts the frame rate of the first camera capturing N video frames at the hardware level based on the degree of content change of M video frames at the software level. By coordinating software and hardware, it improves the acquisition quality of N video frames or reduces acquisition power consumption, thereby enhancing system performance.

[0171] It is understood that the video communication application in Figure 7 above can be a video communication application developed in-house by the terminal manufacturer of the first electronic device. In this case, the first electronic device can acquire M video frames based on the video communication application, and determine a second frame rate for N video frames based on the content variation of the M video frames and the first frame rate, thereby enabling the first camera to acquire N video frames at the second frame rate. It is understood that implementing the method provided in this application based on the video communication application developed in-house by the terminal manufacturer of the first electronic device can ensure the security of parameters such as the acquisition frame rate of the first camera, and can also ensure the compatibility of software and hardware when implementing the method.

[0172] It is understood that, based on the foregoing, the frame rate control method provided in this application is applicable to scenarios including real-time video communication. In the scenario of real-time video communication between a first electronic device and a second electronic device (as shown in Figure 2, a video call), it is understood that the first camera of the first electronic device first captures N video frames corresponding to user A, then the first electronic device (e.g., an encoder in the first electronic device) encodes the N video frames, and then displays the N video frames on the screen, for example, based on the display module in the first electronic device. Furthermore, the first electronic device can also send N video frames at a second frame rate to the second electronic device, so that the second electronic device can display the N video frames at the second frame rate on its screen.

[0173] Figure 9 illustrates the interactive flow of the frame rate control method provided in this application in a video call scenario. As shown in Figure 9, this method can be interactively executed by a first electronic device (mobile phone 100 in Figure 2) and a second electronic device (mobile phone 200 in Figure 2). Specifically, this method includes, but is not limited to, the following schemes:

[0174] 901: Mobile phone 100 and mobile phone 200 establish a video call.

[0175] As you can understand, the scenario of video calling between mobile phone 100 and mobile phone 200 can be seen in Figure 2 above, and will not be repeated here.

[0176] 902: The first camera of mobile phone 100 captures M consecutive video frames at the first frame rate.

[0177] Taking the scenario shown in Figure 2 as an example, after mobile phone 100 and mobile phone 200 establish a video call, the first camera in mobile phone 100 can capture M video frames corresponding to user A at the first frame rate.

[0178] 903: Mobile phone 100 adjusts the first frame rate based on the content change of M video frames to obtain the second frame rate.

[0179] Specifically, mobile phone 100 can obtain a second frame rate based on its internal frame rate determination module. If the degree of content change meets the frame rate adjustment conditions, mobile phone 100 can adjust the first frame rate to obtain the second frame rate. The method by which mobile phone 100 adjusts the first frame rate to obtain the second frame rate is similar to the method described in section 504 above, and will not be repeated here.

[0180] 904: The first camera of mobile phone 100 captures N consecutive video frames at the second frame rate.

[0181] The second frame rate is different from the first frame rate, and the first video frame out of N video frames is displayed after the Mth video frame out of M video frames has been displayed.

[0182] In the embodiments of this application, M and N are integers greater than 1, and M and N can be the same or different, which is not limited in this embodiment.

[0183] In some embodiments, the first camera captures N video frames at a second frame rate, meaning the first camera captures N video frames at a rate of N video frames per second (the number of video frames captured at the second frame rate). For example, if the second frame rate is 15 FPS, then the first camera of the mobile phone 100 can capture N video frames at a rate of 15 video frames per second.

[0184] 905: Mobile phone 100 encodes and displays N consecutive video frames.

[0185] In this embodiment, the mobile phone 100 can encode N video frames based on its internal encoder, and then display the encoded N video frames based on its internal display module. In some embodiments, the mobile phone 100 encodes N video frames at a second frame rate, meaning that the mobile phone 100 encodes N video frames at a rate of [number] video frames per second. For example, if the second frame rate is 15 FPS, then the mobile phone 100 can encode N video frames at a rate of 15 video frames per second.

[0186] 906: Mobile phone 100 sends N video frames at the second frame rate to mobile phone 200.

[0187] For example, mobile phone 100 can send a second frame rate and N video frames to mobile phone 200. After receiving the second frame rate and N video frames, mobile phone 200 can display the received N video frames at the second frame rate.

[0188] It is understood that this application embodiment does not limit the order in which mobile phone 100 displays N video frames or sends N video frames to mobile phone 200. For example, mobile phone 100 may encode N video frames and then send them to mobile phone 200. Alternatively, mobile phone 100 may display N video frames and then send them to mobile phone 200.

[0189] 907: Mobile phone 200 displays the received N video frames at the second frame rate.

[0190] After receiving N video frames sent by mobile phone 100 at the second frame rate, mobile phone 200 can display the N video frames on the screen at the second frame rate. In this way, it can be ensured that the frame rates of mobile phone 100 and mobile phone 200 are adjusted synchronously when displaying N video frames.

[0191] It is understood that Figure 9 only describes the frame rate control method of this application by taking the adjustment of the acquisition frame rate of N video frames corresponding to user A captured by the first camera in mobile phone 100 and sending the N video frames to mobile phone 200 as an example. However, it should be understood that during a video call, the acquisition frame rate of N video frames corresponding to user B captured by the second camera in mobile phone 200 can also be adjusted and sent to mobile phone 100. The method of adjusting the acquisition frame rate of N video frames corresponding to user B captured by the second camera in mobile phone 200 and sending the N video frames to mobile phone 100 is similar in principle to the methods described in 901 to 907 above, and will not be repeated here.

[0192] Compared to real-time video communication scenarios such as video calls, where the first camera of the first electronic device uses a fixed frame rate throughout the entire video call, the method provided in this application allows the first electronic device (such as a mobile phone 100) to adjust the frame rate of the first camera for N video frames at the hardware level based on the degree of content change of M video frames at the software level. This enables the mobile phone 100 to simultaneously adjust the display frame rate of the N video frames. This approach improves system performance by coordinating software and hardware to enhance the display quality of the N video frames or reduce the power consumption of acquiring the N video frames.

[0193] Furthermore, the first electronic device can also send the second frame rate to the second electronic device (such as mobile phone 200), so that the display frame rate of the N video frames of the second electronic device can also be synchronously changed to the second frame rate. In this way, the display quality of the N video frames of the second electronic device can be improved, or the power consumption of the second electronic device when displaying N video frames can be reduced.

[0194] In other embodiments, besides directly determining the second frame rate based on the content change degree of M video frames, the first electronic device can also determine the second frame rate based on the content change degree, the first network information of the network it is in, and / or the first temperature of the first electronic device. Figure 10 shows a flowchart illustrating a method for determining the acquisition frame rate based on the content change degree, the first network information, and the first temperature of the first electronic device. As shown in Figure 10, this method can be executed by the first electronic device. Specifically, this method includes, but is not limited to, the following schemes:

[0195] 1001: Determine the third frame rate of N video frames based on the degree of content change of M consecutive video frames.

[0196] It is understandable that if the degree of content change is greater than the first degree of change threshold and falls within the first degree of content change range, then the frame rate corresponding to the first degree of content change range will be used as the third frame rate.

[0197] 1002: The predicted transmission bitrate for N video frames is obtained based on the first network information corresponding to M consecutive video frames.

[0198] In this embodiment of the application, the first network information includes, but is not limited to, the transmission bitrate, reception bitrate, latency, jitter, and packet loss information corresponding to the M video frames during the transmission process.

[0199] In some embodiments, the first electronic device can determine the transmission bitrate corresponding to N video frames by real-time monitoring of first network information based on a congestion control algorithm. For example, if a large packet loss rate is detected in the first network information, the first electronic device can reduce the transmission bitrate corresponding to N video frames based on the congestion control algorithm. As another example, if a large delay is detected in the first network information, the first electronic device can also reduce the transmission bitrate corresponding to N video frames based on the congestion control algorithm to avoid congestion during transmission.

[0200] In other embodiments, the first electronic device may also input the first network information corresponding to M video frames into a pre-trained transmission bitrate prediction model to obtain the predicted transmission bitrate corresponding to N video frames.

[0201] It is understood that the training set corresponding to the pre-trained transmit bitrate determination model includes network information for the previous time interval (e.g., transmit bitrate, receive bitrate, latency, jitter, packet loss, etc.) and a reference transmit bitrate for the next time interval. During training, the network information used for training is input into the model to obtain the predicted transmit bitrate for the next time interval. Then, the model is repeatedly trained based on the difference between the predicted transmit bitrate and the reference transmit bitrate until a fully trained transmit bitrate determination model is obtained. After obtaining the transmit bitrate determination model, the first electronic device can input the first network information corresponding to the above M video frames into the trained transmit bitrate determination model to obtain the transmit bitrates corresponding to the N video frames output by the model.

[0202] 1003: Based on the transmission bitrate, look up the code table to obtain the fourth frame rate corresponding to the transmission bitrate.

[0203] In an exemplary embodiment, the code table includes a correspondence between multiple transmission bitrates and multiple frame rates. After obtaining the transmission bitrates corresponding to N video frames, the first electronic device can, based on the code table, obtain the frame rate corresponding to the transmission bitrates of the N video frames in the code table, and use that frame rate as the fourth frame rate.

[0204] 1004: Based on the first temperature of the first electronic device corresponding to M consecutive video frames, the second temperature of the first electronic device corresponding to N video frames is predicted.

[0205] In this embodiment of the application, the first electronic device can input the first temperature corresponding to M video frames into a pre-trained device temperature determination model to obtain the predicted second temperature corresponding to N video frames.

[0206] It is understood that the training set for the pre-trained device temperature determination model includes the device temperature corresponding to the previous time interval used for training, and the reference device temperature corresponding to the next time interval. During training, the device temperature corresponding to the previous time interval used for training is input into the model to obtain the predicted device temperature for the next time interval. Then, the model is repeatedly trained based on the difference between the predicted device temperature and the reference device temperature until a fully trained device temperature determination model is obtained. Then, the first electronic device can input the first temperature corresponding to the above M video frames into the trained device temperature determination model to obtain the second temperature corresponding to the N video frames output by the model.

[0207] 1005: Based on the second temperature and the first correspondence, the fifth frame rate corresponding to the second temperature is obtained.

[0208] In an exemplary embodiment, the first correspondence is used to represent the correspondence between multiple device temperatures and multiple frame rates. After obtaining the second temperature corresponding to N video frames, the first electronic device can obtain the fifth frame rate corresponding to the second temperature based on the first correspondence. It can be understood that the second temperature and the fifth frame rate can be negatively correlated, that is, the higher the second temperature, the lower the fifth frame rate. In this way, when the second temperature is high, the frame rate can be reduced, thereby reducing the device temperature and avoiding the problem of the first electronic device's performance degrading due to excessively high device temperature.

[0209] It is understood that the embodiments of this application do not limit the execution order of 1001, 1002 and 1004 above, as long as the first electronic device can determine the third frame rate, the fourth frame rate and the fifth frame rate.

[0210] 1006: When the third frame rate is less than the fourth frame rate, or when the third frame rate is less than the fifth frame rate, the third frame rate will be used as the second frame rate.

[0211] 1007: The first camera of the first electronic device captures N video frames at the second frame rate.

[0212] The specific method by which the first camera of the first electronic device captures N video frames at the second frame rate has been described in detail in section 904 above, and will not be repeated here.

[0213] The method provided in this application comprehensively determines the acquisition frame rate (such as the second frame rate) based on a third frame rate corresponding to the degree of content change, a fourth frame rate corresponding to the first network information, and a fifth frame rate corresponding to the first temperature. It then acquires N video frames based on the determined acquisition frame rate from the first camera, which can reduce the power consumption corresponding to N video frames. Furthermore, compared to methods that determine the acquisition frame rate solely based on network information and device temperature, this application also determines the acquisition frame rate based on the degree of content change, which results in higher accuracy in determining the acquisition frame rate.

[0214] Figure 11 illustrates a flowchart of a method for determining the second frame rate. As shown in Figure 11, the first electronic device can train the decision tree model based on the training set corresponding to the decision tree model described above, obtain a trained decision tree model, and deploy the trained decision tree model in the decision engine. Thus, the encoder in the first electronic device can input the content information of the M video frames (such as the first MSE, first MADP, first I-frame ratio, first texture difference mean, and first QP difference) determined during the encoding of M video frames into the decision tree model in the decision engine to obtain the third frame rate corresponding to the degree of content change represented by the above content information. Furthermore, the first electronic device can also input the first network information corresponding to the M video frames and the first temperature of the first electronic device into the decision engine, and the decision engine obtains the fourth frame rate corresponding to the first network information and the fifth frame rate corresponding to the first temperature. Then, if the third frame rate is less than the fourth and fifth frame rates, the third frame rate is used as the second frame rate and output as the second frame rate.

[0215] In some embodiments, this application also provides a computer-readable medium storing instructions that, when executed on a computer, cause the computer to perform the frame rate control method described in the above embodiments.

[0216] In some embodiments, this application also provides an electronic device, which includes: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the frame rate control method described in the above embodiments.

[0217] In some embodiments, this application also provides a computer program product, including: computer instructions, which, when executed on an electronic device, cause the electronic device to implement the frame rate control method described in the above embodiments.

[0218] Figure 12 shows a schematic diagram of the structure of the electronic device provided in an embodiment of this application. As shown in Figure 12, the electronic device (e.g., a first electronic device or a second electronic device) may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0219] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0220] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0221] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0222] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. The processor can be used to execute the frame rate control method mentioned in this application.

[0223] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0224] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory disposed within the processor.

[0225] The SIM card interface 195 is used to connect the SIM card.

[0226] It is understood that, as used herein, the term “module” may refer to or include, or be part of, an application-specific integrated circuit (ASIC), electronic circuitry, a processor (shared, dedicated, or grouped) and / or memory that executes one or more software or firmware programs, combinational logic circuitry, and / or other suitable hardware components that provide the described functionality.

[0227] It is understood that in the various embodiments of this application, the processor may be a microprocessor, a digital signal processor, a microcontroller, etc., and / or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, etc., and / or any combination thereof.

[0228] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0229] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0230] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0231] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0232] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0233] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0234] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0235] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A frame rate control method, characterized in that, Applied to a first electronic device, the first electronic device including a first camera, the method includes: The first camera captures M consecutive video frames at a first frame rate; The first electronic device determines that the content changes of the M video frames meet the frame rate adjustment conditions; The first camera captures N consecutive video frames at a second frame rate; Wherein, the second frame rate is different from the first frame rate, and the first video frame among the N video frames is acquired after the Mth video frame among the M video frames is acquired, and M and N are integers greater than 1, and M and N may be the same or different.

2. The method according to claim 1, characterized in that, The second frame rate is greater than the first frame rate, and the frame rate adjustment condition is that the content change of the M video frames is greater than the first change threshold.

3. The method according to claim 1, characterized in that, The second frame rate is less than the first frame rate, and the frame rate adjustment condition is that the content change of the M video frames is less than the second change threshold.

4. The method according to claim 2 or 3, characterized in that, The degree of content variation of the M video frames is determined based on at least one of the following: the first mean square error of the M video frames, the first mean absolute difference of pixels, the first mean texture difference, the first keyframe ratio, and the first quantization parameter difference.

5. The method according to claim 1, characterized in that, The first electronic device determines that the content changes of the M video frames satisfy the frame rate adjustment conditions, including: The first electronic device determines, based on a decision tree model, that the degree of content change in the M video frames satisfies the frame rate adjustment condition, wherein the decision tree model is trained based on a classification and regression tree algorithm.

6. The method according to claim 1, characterized in that, The first electronic device determines the acquisition frame rate of the N video frames to be the second frame rate in the following way: Determine that the content change degree of the M video frames belongs to the first content change degree range, and obtain the third frame rate corresponding to the first content change degree range; Obtain the fourth frame rate corresponding to the first network information corresponding to the M video frames, and / or the fifth frame rate corresponding to the first temperature of the first electronic device corresponding to the M video frames; If the third frame rate is less than the fourth frame rate, and / or the third frame rate is less than the fifth frame rate, the third frame rate shall be used as the second frame rate.

7. The method according to claim 1, characterized in that, The method further includes: After the first camera captures N consecutive video frames at the second frame rate, the first electronic device encodes and displays the N video frames at the second frame rate.

8. The method according to claim 1, characterized in that, The method further includes: Corresponding to the real-time video communication between the first electronic device and the second electronic device, the N video frames at the second frame rate are sent to the second electronic device, wherein the second electronic device is used to display the N video frames at the second frame rate.

9. An electronic device, characterized in that, include: One or more processors; One or more memories; the one or more memories storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the frame rate control method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the frame rate control method according to any one of claims 1 to 8.

11. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the frame rate control method according to any one of claims 1 to 8.

Images