Image processing method and apparatus, electronic device, storage medium, and program product

Abstract

An image processing method, comprising: in response to a zoom operation, acquiring a current zoom ratio corresponding to a current image frame and a target zoom ratio issued by a camera application, wherein the current zoom ratio is used for determining optical zoom point identification information, and the target zoom ratio is used for determining zoom speed information; determining a target frame interpolation count on the basis of the optical zoom point identification information and the zoom speed information; and using a zoom ratio of the target frame interpolation count to process a target image.

Description

Image processing methods, apparatuses, electronic devices, storage media, and program products

[0001] Cross-reference of related applications

[0002] This application claims priority to Chinese Patent Application No. 2024119683738, filed on December 27, 2024, entitled "Image Processing Method, Apparatus, Electronic Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0004] With the development of image processing technology, electronic devices are equipped with multiple cameras corresponding to different focal lengths. Each focal length has a different focal length range, and therefore a different field of view. Through image zoom technology, clear images can be captured at different distances.

[0005] However, during zooming, the image smoothness is low due to camera switching. Summary of the Invention

[0006] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium that can improve the smoothness of image switching during zooming.

[0007] In a first aspect, this application provides an image processing method, comprising:

[0008] In response to a zoom operation, the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application are obtained; wherein, the current zoom ratio is used to determine the optical zoom point identification information, and the target zoom ratio is used to determine the zoom speed information;

[0009] The number of target frames is determined based on the optical zoom point identification information and the zoom speed information; and

[0010] The image is processed using the zoom ratio of the target number of interpolated frames.

[0011] Secondly, this application also provides an image processing apparatus, comprising:

[0012] The acquisition module is used to acquire the current zoom ratio and the target zoom ratio issued by the camera application in response to the zoom operation; wherein, the current zoom ratio is used to determine the optical zoom point identification information, and the target zoom ratio is used to determine the zoom speed information.

[0013] The scene module is used to determine the number of target frames to be interpolated based on the optical change point identification information and the zoom speed information; and

[0014] The processing module is used to process the target image using the zoom ratio of the target interpolation frame number.

[0015] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program as an image processing method of the first aspect.

[0016] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method of the first aspect.

[0017] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method of the first aspect.

[0018] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained from these drawings without creative effort.

[0020] Figure 1 is an application environment diagram of an image processing method in one embodiment.

[0021] Figure 2 is a schematic diagram of an image captured in one embodiment.

[0022] Figure 3 is a schematic diagram of an electronic device and the images it captures in one embodiment.

[0023] Figure 4 is a flowchart illustrating an image processing method in one embodiment.

[0024] Figure 5 is a flowchart illustrating the process of determining the target number of interpolated frames in one embodiment.

[0025] Figure 6 is a schematic diagram of the frame interpolation curve in one embodiment.

[0026] Figure 7 is a schematic diagram of the overall process in a sliding zoom scenario in one embodiment.

[0027] Figure 8 is a schematic diagram of the zoom magnification in a sliding zoom scene in one embodiment.

[0028] Figure 9 is a schematic diagram of a frame interpolation strategy in one embodiment.

[0029] Figure 10 is a schematic diagram of the tail interpolation curve in one embodiment.

[0030] Figure 11 is a structural block diagram of an image processing device in one embodiment.

[0031] Figure 12 is an internal structure diagram of an electronic device in one embodiment. Detailed Implementation

[0032] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of this application.

[0033] Some electronic devices may have 3-5 different types of lenses. For example, the camera operation page of a multi-camera phone will display focal length selections of 0.6X, 1X, 2X, or 5X, 10X. The smaller the number, the wider the field of view and the farther away the subject is. These roughly correspond to the focal lengths of ultra-wide-angle (UW), wide-angle (W), and telephoto (T) lenses.

[0034] Taking a certain mobile phone as an example, in some situations, the main camera of this phone is a wide-angle lens with a focal length of approximately 28mm. This focal length is close to the field of view of the human eye, and the main camera is the most frequently used camera. The phone's camera operation page will display the value of 1X, as shown in Figure 1. 1X is the focal length corresponding to the main camera, and the image quality is relatively clear, suitable for shooting portraits, architecture, landscapes, and documentary photography.

[0035] Ultra-wide-angle lenses offer a wider field of view than wide-angle lenses, capturing a much broader scene. Some wide-angle lenses are indicated by their zoom ratio on the control panel, such as 0.5X or 0.6X; others use images or text. Ultra-wide-angle lenses can include more scenic elements in the frame, making them suitable for landscape and architectural photography, producing images with a powerful visual impact. Ultra-wide-angle lenses exhibit lens distortion, elongating and magnifying objects at the edges of the image. This distortion can be fully utilized for shooting from a low angle, creating a visually striking "nearer objects appear larger, farther objects smaller" effect. When photographing buildings, lens distortion can be used to make buildings appear even more imposing. An image captured by an ultra-wide-angle lens is shown in Figure 2.

[0036] In addition to wide-angle and ultra-wide-angle lenses, mobile phone cameras can also include telephoto lenses. Telephoto lenses are used to capture images with a zoom ratio of 1X or greater. The larger the number before X, the narrower the shooting range, but the farther the subject can be captured, and the sharper the distant objects are. When shooting distant objects or magnifying objects in the scene using a telephoto lens, the image quality does not degrade like with digital zoom. In situations where movement is inconvenient, such as framing a shot in a cluttered building, it's difficult to capture the subject using only the main camera. Using a telephoto lens in this case can add more depth to the image. Telephoto lenses can "bring in" the distance between the background and foreground, creating a sense of spatial compression and making the overall image fuller. Telephoto lenses have less distortion and weaker perspective effects, which can bring the foreground and background closer, enhancing the relationship between them and creating unique visual effects. This feature can be used to guide the viewer's attention back to the subject in the depth of the image by using straight lines such as roads and railings.

[0037] Lenses with different focal lengths have different characteristics. Camera applications are designed with Spatial Alignment Transform (SAT) to switch lenses in real time according to user needs in different shooting scenarios. For example, a mobile phone uses a wide-angle lens with an equivalent focal length of 27mm, 40-megapixel resolution, and an aperture with a focal ratio of 1.6; an ultra-wide-angle lens with an equivalent focal length of 16mm and an aperture with a focal ratio of 2.2; and a telephoto lens with an equivalent focal length of 125mm and an aperture with a focal ratio of 3.4, which can continuously change the focal length from 16 to 27mm and from 27 to 125mm.

[0038] The electronic device shown in Figure 3(a) includes three cameras: an ultra-wide-angle lens, a wide-angle lens, and a telephoto lens. The zoom level at which the three cameras switch is the optical zoom point. The optical zoom point between the ultra-wide-angle lens and the wide-angle lens is 1X, as shown in Figure 3(b); the wide-angle lens captures images at a zoom level of 2X, as shown in Figure 3(c); the optical zoom point between the wide-angle lens and the telephoto lens is 3X, as shown in Figure 3(d); and the telephoto lens captures images at a zoom level of 5X, as shown in Figure 3(e).

[0039] During zooming, ensuring smooth lens switching and minimizing image jumps have become key considerations for modern cameras. Smoothness refers to the fluidity of the image frames displayed on the screen during zooming, i.e., continuous flow; while minimizing image jumps is achieved through spatial alignment processing to ensure consistent image quality before and after the zoom transition.

[0040] There are various zoom methods. Taking the sliding zoom scenario as an example, the image smoothness in this scenario is relatively poor. Because sliding zoom scenarios have random direction and speed, they require high precision in starting and ending touch input. Furthermore, the limitations of power consumption and platform resources must be considered. Therefore, a balance needs to be struck as much as possible, with different focuses for different sliding zoom scenario characteristics. Power consumption includes, but is not limited to, the three-camera setup issue. Platform resource limitations include, but are not limited to, the 3TFE issue. The three-camera setup issue involves how to activate three cameras and their corresponding sensors; while the 3TFE issue involves three independent touch feature engines and how to handle different touch tasks.

[0041] Through analysis, the inventors of this application discovered that a more refined scene division is needed for the zoom process, and then differentiated frame interpolation control can be applied to the refined scenes. For example, the zoom-in scene focuses on stopping responsiveness and the smoothness of the field of view (FOV) change, while the zoom-out scene, which involves fast zooming and passing through optical zoom points, focuses on the number of interpolated frames to give the target lens time to activate.

[0042] The image processing method provided in this application can be applied to electronic devices. These electronic devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted displays, etc. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The electronic device can be a terminal or a server.

[0043] In some exemplary embodiments, as shown in FIG4, an image processing method is provided for application to an electronic device, the method comprising the following operations 402 to 406, wherein:

[0044] Operation 402, in response to a zoom operation, obtains the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application; wherein, the current zoom ratio is used to determine the optical zoom point identification information, and the target zoom ratio is used to determine the zoom speed information.

[0045] Zooming is an operation used to change the focal length. Zooming can refer to indeterminate continuous zooming, that is, zooming without a specific target zoom ratio. By adjusting the zoom ratio and providing the zoomed image, the user can obtain a better preview or recording image. Zooming can also be a zoom operation that slides to a specific target zoom ratio. Zooming can be implemented through touch or sliding zoom controls, which can also be implemented through zoom controls on the camera application interface. Zoom controls on the camera application interface can be discs, bars, or other shapes. In some possible implementations, zooming can also be achieved by pressing buttons, sliding, or rotating hardware devices, such as configuring a rotatable device on the electronic device, where the zoom ratio is adjusted by rotating the rotatable device; the rotatable device is a component capable of rotation.

[0046] The current image frame is the image frame that the electronic device needs to display at the current moment. The current image frame can be the image frame that needs to be previewed, captured, or recorded for display; correspondingly, zoom operation can be a zoom operation in a camera preview scene, a photo capture scene, or a recording scene. The current zoom ratio refers to the zoom ratio of the current image frame.

[0047] The target zoom ratio refers to the zoom ratio generated by the camera application after detecting a zoom operation. The camera application acquires the zoom ratio obtained from the zoom operation in real time or periodically and uses this acquired zoom ratio as the target zoom ratio. In the case of a zoom operation that is a slide cut, the target zoom ratio can be the target zoom ratio currently acquired for the slide cut operation.

[0048] For example, responding to a zoom operation can mean that the electronic device responds to a zoom operation generated by a user sliding a disc; responding to a zoom operation can also mean that the user rotates a hardware device; responding to a zoom operation can also mean that the user taps the screen with two fingers or multiple fingers. The electronic device can obtain the target zoom magnification through the Hardware Abstraction Layer (HAL).

[0049] The optical zoom point identifier refers to the information indicating whether the zoom process, triggered by a zoom operation, passes through an optical zoom point. It represents a dimension of the zoom scene type. The optical zoom point identifier can indicate whether the zoom process passes through an optical zoom point. It can be represented by numbers, letters, or characters. A first identifier indicates that the zoom process passes through the optical zoom point; a second identifier indicates that the zoom process does not pass through the optical zoom point. For example, a first identifier of true indicates that the zoom process passes through the optical zoom point, and a second identifier of false indicates that the zoom process does not pass through the optical zoom point; similarly, a first identifier of 1 indicates that the zoom process passes through the optical zoom point, and a second identifier of 2 indicates that the zoom process does not pass through the optical zoom point.

[0050] Zoom speed information refers to the type of zoom speed during the zoom process triggered by a zoom operation, used to represent the zoom rate range. Zoom speed information is a dimension of zoom scene type. Zoom speed information can represent the type of zoom speed range. Zoom speed information includes at least a first zoom speed type and a second zoom speed type. The zoom speed of the first zoom speed type is lower than that of the second zoom speed type. In this case, the first zoom speed type can be a slow zoom scene, and the second zoom speed type can be a fast zoom scene.

[0051] For example, zoom speed information can be identification data. Zoom speed information can be represented using numbers, letters, or characters. A first flag indicates a first zoom speed type or a slow zoom scene, while a second flag indicates a second zoom speed type or a fast zoom scene. For example, the first flag might be K and the second flag S; another example is 3 and the second flag 4.

[0052] Both zoom levels, whether decreasing or increasing, can determine the optical zoom point identifier based on the current zoom level and the zoom speed information based on the target zoom level, forming two dimensions of the zoom scene type. These two dimensions allow for more detailed analysis of the zoom scene type during the zoom process, facilitating smooth image transitions. Furthermore, when zooming down, the optical zoom point identifier can be determined based on the current zoom level, and the zoom speed information can be determined based on the target zoom level; while when zooming up, other operations can be performed to adapt to the different field-of-view changes for zooming down and zooming up.

[0053] Operation 404: Determine the target frame interpolation number based on the optical zoom point identification information and zoom speed information.

[0054] The target frame interpolation number is the zoom ratio calculated using the zoom ratio interpolation method. This is used to adjust the zoom ratio to make the zoom process smoother. The target frame interpolation number can include the zoom ratio between the current zoom ratio and the target zoom ratio.

[0055] Adjusting the data in either of the two dimensions—optical zoom point identification information and zoom speed information—can change the calculation method for the target number of interpolated frames, resulting in a corresponding zoom magnification interpolation method. The zoom magnification interpolation method can then be used to obtain the zoom magnification for the target number of interpolated frames. This zoom magnification interpolation method can be implemented using a mapping table, various functions, or by adjusting or selecting the zoom curve based on the optical zoom point identification information and zoom speed information to obtain the target number of interpolated frames.

[0056] For example, while keeping the optical zoom point identification information unchanged, the method for determining the number of target interpolated frames is adjusted using the zoom speed information to obtain the corresponding number of target interpolated frames.

[0057] Operation 406 processes the target image using the zoom ratio of the target interpolation frame count.

[0058] The zoom ratio for the target number of interpolated frames includes the transition zoom ratio between the current zoom ratio and the target zoom ratio. The target zoom curve can be used to insert a zoom ratio between the current zoom ratio and the target zoom ratio; the inserted zoom ratio is the transition zoom ratio. The target zoom curve corresponding to the scene dimension can be determined in each zoom curve based on at least one scene dimension, either optical zoom point identification information or zoom speed information. Each zoom curve can be expressed using a function or mapping table; the function can be, for example, a linear function, a power function, an exponential function, etc.

[0059] For example, the acquired target image data is processed at each zoom level based on the target interpolation frame number to obtain target image data at each transition zoom level; the field of view is determined based on the target image data at each transition zoom level; and the electronic device is controlled to display the target image frame at that field of view.

[0060] In the aforementioned image processing method, in response to a zoom operation, the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application are obtained to clarify the initial current zoom ratio and the target zoom ratio to be achieved during the zoom process. Based on the current zoom ratio, optical zoom point identification information is determined, and based on the target zoom ratio, zoom speed information is determined. The optical zoom point identification information indicates whether the zoom process passes through an optical zoom point, and the zoom speed information indicates the zoom rate range. Furthermore, the frame interpolation method during the zoom process is indirectly controlled through the optical zoom point identification information and the zoom speed information. Therefore, the target number of interpolated frames is determined based on the optical zoom point identification information and the zoom speed information, ensuring that the target number of interpolated frames is adaptively adjusted to suit different scenes. Finally, the target image is processed using the zoom ratio of the target number of interpolated frames, causing the field of view of the target image in the zoom scene to change gradually, avoiding abrupt changes in the field of view and achieving smooth switching of the target image during the zoom process.

[0061] In some embodiments, the method of using the current zoom magnification to determine the optical zoom point identification information includes: when there is an optical zoom point between the current zoom magnification and the target zoom magnification, obtaining optical zoom point identification information to indicate that the zoom process passes through an optical zoom point; when there is no optical zoom point between the current zoom magnification and the target zoom magnification, obtaining optical zoom point identification information to indicate that the zoom process does not pass through an optical zoom point.

[0062] The situation where an optical zoom point exists between the current zoom level and the target zoom level refers to the case where there is at least one optical zoom point within the focal length range between the current zoom level and the target zoom level. When there are multiple target zoom levels, if an optical zoom point exists between the current zoom level and the latest obtained target zoom level, optical zoom point identification information is obtained to indicate that the zoom process passes through an optical zoom point; if no optical zoom point exists between the current zoom level and the latest obtained target zoom level, optical zoom point identification information is obtained to indicate that the zoom process does not pass through an optical zoom point.

[0063] In this embodiment, the scene during the zoom process is accurately refined by identifying the optical shift points between the current zoom level and the target zoom level. When passing through optical shift points, multi-frame interpolation can be used to make the image changes smoother.

[0064] In some embodiments, the method of using the target zoom ratio to determine zoom speed information includes: comparing the difference value between different target zoom ratios with a difference threshold; when the difference value is less than the difference threshold, determining the zoom speed information as a first zoom speed type; when the difference value is greater than or equal to the difference threshold, determining the zoom speed information as a second zoom speed type.

[0065] In a zoom-by-slide scenario, the camera application detects the target zoom magnification during the user's zoom-by-slide operation in real time or periodically and sends the target zoom magnification to the hardware abstraction layer. The hardware abstraction layer receives the two adjacent target zoom magnifications and calculates the difference between them. This difference measure the magnitude of change in the target zoom magnification. This difference can be the difference, ratio, or bitwise operation result between adjacent target zoom magnifications. Correspondingly, the difference threshold can be a threshold for the difference, ratio, or bitwise operation result between adjacent target zoom magnifications. The difference threshold can be preset or dynamically adjusted based on the scene. The difference value is compared with the difference threshold. When the difference value is less than the difference threshold, the zoom speed information is determined to be of the first zoom speed type; when the difference value is greater than or equal to the difference threshold, the zoom speed information is determined to be of the second zoom speed type.

[0066] In this embodiment, a difference value is determined based on the separately acquired target zoom ratios, and then this difference value is compared with a difference threshold to perform qualitative analysis more efficiently. That is, when the difference value between different target zoom ratios is small, the camera of the electronic device is in a slow zoom type, and when the difference value between different target zoom ratios is large, the camera of the electronic device is in a fast zoom type.

[0067] In some embodiments, the method of using the current zoom ratio to determine the optical zoom point identification information includes: determining the optical zoom point identification information based on the current zoom ratio when the current zoom ratio is greater than the target zoom ratio.

[0068] The situation where the current zoom ratio is greater than the target zoom ratio refers to the process of reducing the size of the object in the current image frame and increasing the field of view by zooming.

[0069] In some embodiments, when the current zoom ratio is greater than the target zoom ratio, determining the optical zoom point identification information based on the current zoom ratio includes: if the current zoom ratio is greater than the target zoom ratio and there is an optical zoom point between the current zoom ratio and the target zoom ratio, obtaining optical zoom point identification information to indicate that the zoom process passes through an optical zoom point; if the current zoom ratio is greater than the target zoom ratio and there is no optical zoom point between the current zoom ratio and the target zoom ratio, obtaining optical zoom point identification information to indicate that the zoom process does not pass through an optical zoom point. Thus, by using the two comparison results between the current zoom ratio and the target zoom ratio, the optical zoom point identification information is accurately determined.

[0070] In some embodiments, when the current zoom ratio is greater than the target zoom ratio, determining the optical zoom point identification information based on the current zoom ratio includes: when there is an optical zoom point between the current zoom ratio and the target zoom ratio, obtaining optical zoom point identification information to indicate that the zoom process passes through an optical zoom point; and when there is no optical zoom point between the current zoom ratio and the target zoom ratio, obtaining optical zoom point identification information to indicate that the zoom process does not pass through an optical zoom point. Thus, the two comparison results between the current zoom ratio and the target zoom ratio are determined sequentially to accurately and systematically determine the optical zoom point identification information.

[0071] In this embodiment, when the current zoom ratio is greater than the target zoom ratio, the field of view gradually increases. Therefore, the information reflected in the current image frame is less than that reflected in the image frame at the target zoom ratio. In this case, it is necessary to determine the optical zoom point identification information based on the current zoom ratio in order to accurately determine whether to increase the number of interpolated frames based on the optical zoom point identification information, thereby reserving opening time for the camera to be switched.

[0072] In some embodiments, the method further includes: obtaining an initial number of interpolated frames when the current zoom ratio is less than the target zoom ratio; and processing the image using the zoom ratio of the initial number of interpolated frames.

[0073] The situation where the current zoom ratio is less than the target zoom ratio refers to the process of increasing the size of objects in the current image frame and decreasing the field of view through zooming. In this case, even if the secondary camera does not zoom in promptly, the FOV is sufficient to support cropping, thus ensuring image smoothness even if the optical zoom point detection is delayed or misjudged.

[0074] The initial number of interpolated frames is the number of interpolated frames related to the zoom speed type. The initial number of interpolated frames includes at least the first initial number of interpolated frames and the second initial number of interpolated frames. The first zoom speed type belongs to the slow zoom scene, while the second zoom speed type belongs to the fast zoom scene. The first initial number of interpolated frames is the number of interpolated frames under the first zoom speed type, and the second initial number of interpolated frames is the number of interpolated frames under the second zoom speed type.

[0075] For example, the acquired image data is processed at each zoom level based on the initial number of interpolated frames to obtain image data at each transition zoom level; the field of view is determined based on the image data at each transition zoom level; and the electronic device is controlled to display the image frame at that field of view.

[0076] In this embodiment, when the current zoom ratio is less than the target zoom ratio, the field of view gradually decreases, so the current image frame reflects more information than the image frame at the target zoom ratio. In this case, image acquisition can be performed through digital zoom to achieve smoothness in the corresponding image zoom process.

[0077] In some embodiments, as shown in FIG5, the determination of the target frame interpolation number in operation 406 based on the optical change point identification information and zoom speed information includes:

[0078] Operation 502: When the optical zoom point identification information indicates that the zoom process does not pass through the optical zoom point, determine the target number of interpolated frames based on the number of interpolated frames corresponding to the zoom speed information.

[0079] The number of interpolated frames corresponding to zoom speed information is the number of interpolated frames adjusted based on zoom speed information; the number of interpolated frames corresponding to zoom speed information can be the actual number or the number represented by zoom speed information.

[0080] For example, zoom speed information can be mapped to obtain the number of interpolated frames corresponding to the zoom speed information; alternatively, the calculation method for the target number of interpolated frames can be adjusted based on the zoom speed information to obtain the zoom ratio interpolation method corresponding to the zoom speed; then, the zoom ratio of the target number of interpolated frames can be obtained through the zoom ratio interpolation method corresponding to the zoom speed. In this case, the number of interpolated frames corresponding to the zoom speed information is represented by the zoom speed information and is the number of interpolated frames obtained by adjusting the zoom ratio interpolation method corresponding to the zoom speed.

[0081] Operation 504: When the optical zoom point identification information indicates that the zoom process passes through the optical zoom point, obtain the interpolation increment corresponding to the optical zoom point, and obtain the target number of interpolation frames based on the interpolation increment and the number of interpolation frames corresponding to the zoom speed information.

[0082] The interpolation increment corresponding to the optical change point is the data used to increase the number of interpolated frames. The interpolation increment can be an actual number, a parameter used to adjust the calculation method of the target number of interpolated frames, or a preset calculation method.

[0083] For example, the electronic device can obtain the interpolation increment corresponding to the optical zoom point according to the mapping table, and add the interpolation increment corresponding to the optical zoom point and the number of interpolated frames corresponding to the zoom speed information to obtain the target number of interpolated frames. Alternatively, the electronic device can adjust the zoom ratio interpolation method corresponding to the zoom speed information based on the interpolation increment corresponding to the optical zoom point to obtain the zoom ratio interpolation method corresponding to the optical zoom point; then, it can obtain the zoom ratio for the target number of interpolated frames through the zoom ratio interpolation method corresponding to the optical zoom point. In this case, the interpolation increment corresponding to the optical zoom point is a parameter used to adjust the calculation method for the target number of interpolated frames.

[0084] Furthermore, when the interpolation increment corresponding to the light change point is calculated using a preset method, the interpolation increment corresponding to the light change point and the number of interpolated frames corresponding to the zoom speed information are functions of the zoom curve under the corresponding scene type, and the target number of interpolated frames is obtained using the function of the zoom curve.

[0085] In this embodiment, when the zoom process does not pass through an optical zoom point, the optical zoom point identification information does not affect the target interpolation frame count. The target interpolation frame count is determined solely based on the interpolation frame count corresponding to the zoom speed information to ensure processing efficiency and avoid excessive power consumption due to the secondary camera pulling up the focal length for too long, thus achieving a balance between smoothness and power consumption. When the zoom process passes through an optical zoom point, a focal length switch is required. Therefore, the interpolation frame increment corresponding to the optical zoom point increases the interpolation frame count corresponding to the zoom speed information. On the one hand, this allows the camera that is not activated and is waiting to be switched sufficient startup time to ensure a smoother switching process between different image frames. On the other hand, it improves the alignment and fusion of images captured by different cameras, resulting in a smoother image frame transition.

[0086] In some embodiments, the method for determining the number of interpolated frames corresponding to the zoom speed information includes: when the zoom speed information represents a first zoom speed type, obtaining a first initial number of interpolated frames, and obtaining a target number of interpolated frames based on the first initial number of interpolated frames; when the zoom speed information represents a second zoom speed type, obtaining a second initial number of interpolated frames, and obtaining a target number of interpolated frames based on the second initial number of interpolated frames; wherein the first initial number of interpolated frames is less than the second initial number of interpolated frames; and the zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type.

[0087] The first zoom speed type and the second zoom speed type are different zoom speed information. The zoom speed corresponding to the first zoom speed type is lower than that corresponding to the second zoom speed type. Therefore, the first zoom speed type belongs to a slow zoom scenario, while the second zoom speed type belongs to a fast zoom scenario. For example, when the zoom operation is a slide-cut operation, the first zoom speed type can be a slow slide zoom scenario, and the second zoom speed type can be a fast slide zoom scenario.

[0088] The first initial frame interpolation number is the number of frames interpolated under the first zoom speed type, and the second initial frame interpolation number is the number of frames interpolated under the second zoom speed type. The first initial frame interpolation number can be determined based on a function of the first zoom curve, and the second initial frame interpolation number can be determined based on a function of the second zoom curve. Alternatively, both the first and second initial frame interpolation numbers can be determined based on a mapping table; the first initial frame interpolation number in the mapping table can fit the first zoom curve, and the second initial frame interpolation number can fit the second zoom curve. For example, the first zoom curve is a curve represented by an exponential function, and the second zoom curve is a curve represented by a linear function.

[0089] Since the first initial number of interpolated frames is less than the second initial number of interpolated frames, the number of interpolated frames obtained based on the first initial number of interpolated frames is less than the number of interpolated frames obtained based on the second initial number of interpolated frames.

[0090] Specifically, when the zoom operation belongs to the first zoom speed type, the first initial frame interpolation number can be used as the frame interpolation number corresponding to the zoom speed information; alternatively, the first initial frame interpolation number can be filtered based on other parameters to select the frame interpolation number corresponding to the zoom speed information. When the zoom operation belongs to the second zoom speed type, the second initial frame interpolation number can be used as the frame interpolation number corresponding to the zoom speed information; alternatively, the second initial frame interpolation number can be filtered based on other parameters to select the frame interpolation number corresponding to the zoom speed information.

[0091] In this embodiment, there are at least two zoom speed types. For each zoom speed type, the zoom speed and the initial number of interpolated frames are positively correlated. Therefore, when the zoom speed is relatively slow, the initial number of interpolated frames is relatively small. This not only allows for better alignment and fusion of images captured by different cameras to achieve better motion effect changes, but also avoids wasting computational power, ensuring smoother image frames. Conversely, when the zoom speed is relatively fast, the number of target interpolated frames is larger, resulting in more sufficient startup time for the target camera to be switched, ensuring a smoother switching process between different image frames.

[0092] In some embodiments, in the zoom ratio sequence corresponding to the second initial number of interpolated frames, the difference between adjacent zoom ratios in the first N zoom ratios is less than the difference between adjacent zoom ratios in the zoom ratios after the Nth zoom ratio.

[0093] A zoom ratio sequence is a sequence of zoom ratios arranged in time. During a zoom operation, the first frame of the zoom ratio sequence can be the current zoom ratio, or it can be a zoom ratio determined by frame interpolation after the current zoom ratio; the last frame of the zoom ratio sequence can be the target zoom ratio, or it can be the last zoom ratio determined by frame interpolation before the target zoom ratio.

[0094] N is a positive integer greater than or equal to 2; the first N zoom levels include at least two zoom levels, thus the concept of adjacent zoom levels exists; adjacent zoom levels can be any two zoom levels at adjacent times. Zoom levels after the Nth can include the Nth zoom level, and can also include the (N+M)th zoom level, where M is a positive integer. The difference between adjacent zoom levels can be the numerical difference between the zoom levels, or it can be a normalized difference.

[0095] For example, a slow-out process is performed in the first two frames, meaning the zoom ratio changes more gradually, and then a large-span mapping is performed after two frames. When the zoom ratio is reduced (zoom in), the interpolation curve formed by the zoom ratio and the number of image frames is shown in Figure 6(a), such as the 14th image frame corresponding to a zoom ratio of 3X; when the zoom ratio is increased (zoom out), the interpolation curve formed by the zoom ratio and the number of image frames is shown in Figure 6(b), such as the 14th image frame corresponding to a zoom ratio of 1X.

[0096] In this embodiment, during the initial stage of rapid zoom, the zoom ratio changes relatively slowly, allowing for a smoother change in the image field of view. After the initial stage of rapid zoom, the zoom ratio is adjusted rapidly to reach the target zoom ratio more quickly, thereby ensuring high efficiency in adjusting the zoom ratio.

[0097] In some embodiments, the method further includes: determining a target optical zoom point based on the current zoom ratio and the target zoom ratio, when the zoom speed information represents a second zoom speed type; determining the camera to be switched based on the target optical zoom point; and controlling the camera to be switched to be in an on state.

[0098] The camera to be switched to includes an image sensor. The camera to be switched to includes at least a target camera with a focal length that includes the target zoom ratio; and if the zoom process passes through an optical zoom point, the camera to be switched to may also include an intermediate camera with a focal length between the current camera and the target camera.

[0099] The target optical zoom point is the optical zoom point that needs to be passed through during the zoom process. The target optical zoom point can be determined based on the focal length between the current zoom ratio and the target zoom ratio; alternatively, it can be determined based on the difference or ratio between the current zoom ratio and the target zoom ratio.

[0100] Controlling the camera to be switched to be in an on state includes at least turning on the camera to be switched to that is in an off state; it may also include keeping the camera to be switched to that is in an on state.

[0101] For example, the camera to be switched can be determined based on the latest target zoom ratio and the current zoom ratio, and the cameras passed through during the zoom process and the target camera at the final target zoom ratio can be selected as the cameras to be switched, while keeping these cameras in an on state. Furthermore, if these cameras are already on, their state remains unchanged.

[0102] In this embodiment, since the zoom rate is fast and the target zoom magnification may change in the fast zoom scene, the target optical change point is determined in advance. The camera to be switched is determined by the optical change point, so that the start node of the camera to be switched is earlier than the current image frame display node of the electronic device, thereby reducing the startup time and ensuring a smoother switching process between different image frames.

[0103] In some embodiments, obtaining the interpolation increment corresponding to the optical zoom point includes: obtaining the first interpolation increment corresponding to the optical zoom point when the zoom speed information represents a first zoom speed type; obtaining the second interpolation increment corresponding to the optical zoom point when the zoom speed information represents a second zoom speed type; wherein the number of interpolated frames increased by the first interpolation increment is greater than the number of interpolated frames increased by the second interpolation increment.

[0104] The first frame interpolation increment is the frame interpolation increment under the first zoom speed type. The target number of interpolated frames under the first zoom speed type can be determined by the first frame interpolation increment and the first initial number of interpolated frames. The second frame interpolation increment is the frame interpolation increment under the second zoom speed type. The target number of interpolated frames under the second zoom speed type can be determined by the second frame interpolation increment and the second initial number of interpolated frames.

[0105] Since the number of interpolated frames increased by the first interpolation increment is greater than the number of interpolated frames increased by the second interpolation increment, when the zoom process passes through the optical zoom point, the magnitude of the adjustment of the first initial interpolation frame number by the first interpolation increment is greater than the magnitude of the adjustment of the second initial interpolation frame number by the second interpolation increment.

[0106] For example, regardless of whether the zoom level is in the zoom-in or zoom-out scene, when the slow zoom process passes through the optical change point, 3 frames are added through the first frame interpolation increment to increase the smoothing effect; regardless of whether the zoom level is in the zoom-in or zoom-out scene, when the fast zoom process passes through the optical change point, 1 frame is added through the second frame interpolation increment to wait for the camera to be switched to enter the on state.

[0107] In this embodiment, when the zoom process passes through an optical change point and the zoom is slow, the number of interpolated frames increased by the first interpolation increment is relatively large, resulting in better alignment and fusion animation effects, so that the zoom process information of the image frame can be displayed more meticulously; when the zoom process passes through an optical change point and the zoom is slow, the number of interpolated frames increased by the second interpolation increment is relatively small, so that the reserved time for the camera to be switched to enter the on state is more sufficient.

[0108] In some embodiments, after processing the target image using the zoom ratio of the target number of interpolated frames, the method further includes: determining the number of tail interpolated frames corresponding to the end stage of the zoom process when the continuous and matching target zoom ratios reach a preset number; processing the tail image using the zoom ratio of the tail interpolated frames; and the acquisition time of the tail image is later than the acquisition time of the target image.

[0109] The zoom-to-end phase is the stage where the zoom process stops. At this phase, there is no new target zoom magnification, so smoothing can be achieved through additional frame interpolation strategies. For example, a frame interpolation curve can be used to perform end-of-pipe frame interpolation, obtaining the number of end-of-pipe frames to achieve end-of-pipe frame interpolation.

[0110] The number of frames inserted at the end of the zoom process refers to the number of frames inserted at the end of the zoom process, where no new target zoom magnification was detected. For example, the zoom magnification of the number of frames inserted at the end is smoothed and eased according to the tail curve compared to the beginning of the zoom process or other non-initial phases.

[0111] The tail image is acquired and processed after the target image. The target image and the tail image are processed using different zoom levels, resulting in different zoom speeds for the target image and the tail image.

[0112] Achieving a preset target zoom ratio requires at least two characteristics: continuity and matching. Continuity means that the zoom ratios are detected by the electronic device at adjacent moments, while matching means that they are equal or the difference is less than the preset target zoom ratio difference. For example, the camera application may send three consecutive target zoom ratios of 1X, 1X, and 1X.

[0113] In some embodiments, determining the number of trailing frames corresponding to the end stage of the zoom process includes: obtaining an initial number of trailing frames when the optical change point identification information indicates that the zoom process does not pass through an optical change point, and obtaining the number of trailing frames based on the initial number of trailing frames.

[0114] For example, the acquired tail image data is processed based on each zoom level of the number of tail interpolation frames to obtain tail image data at each transition zoom level; the field of view is determined based on the tail image data at each transition zoom level; and the electronic device is controlled to display the tail image frame at that field of view.

[0115] In this embodiment, the number of consecutive and matching target zoom ratios is counted. Based on the comparison between the count results and the preset number, the end stage of the zoom process is determined. The frame interpolation strategy of the zoom ratio is changed accordingly. The processing efficiency is ensured through the reuse process of the target zoom ratio.

[0116] In some embodiments, after processing the target image using the zoom magnification of the target number of interpolated frames, the method further includes: determining the number of tail interpolated frames corresponding to the end stage of the zoom process when a stop event flag is detected; processing the tail image using the zoom magnification of the tail interpolated frames; wherein the acquisition time of the tail image is later than the acquisition time of the target image.

[0117] A stop event flag is data used to indicate the end of the zoom process. A stop event flag can be, but is not limited to, an identifier, comparison result, or the result of event processing.

[0118] The tail image is acquired and processed after the target image. The target image and the tail image are processed using different zoom levels, resulting in different zoom speeds for the target image and the tail image.

[0119] For example, the acquired tail image data is processed based on each zoom level of the number of tail interpolation frames to obtain tail image data at each transition zoom level; the field of view is determined based on the tail image data at each transition zoom level; and the electronic device is controlled to display the tail image frame at that field of view.

[0120] In this embodiment, the end stage of the zoom process is accurately determined by a stop event flag, thereby changing the frame interpolation strategy of the zoom magnification. The stop event flag also ensures the accuracy of the identification of the end stage of the zoom process.

[0121] In some embodiments, when the optical change point identification information indicates that the zoom process passes through an optical change point, determining the number of tail interpolated frames corresponding to the end stage of the zoom process includes: obtaining the initial number of tail interpolated frames and the tail interpolated frame increment corresponding to the optical change point, and obtaining the number of tail interpolated frames based on the tail interpolated frame increment corresponding to the optical change point and the initial number of tail interpolated frames.

[0122] The initial number of tail frames can be the number of frames inserted at the end of the zoom process, assuming the zoom process does not pass through the optical change point, representing the zoom magnification at the end of the zoom process. The initial number of tail frames can be used to create a gradual fade-out effect between the tail image and the target image. The initial number of tail frames can be determined as a function of the tail curve.

[0123] The tail frame interpolation increment is the data used to increase the number of interpolated frames. The tail frame interpolation increment can be the same data as the interpolation increment, or it can be different data. The tail frame interpolation increment can be an actual number, a parameter used to adjust the calculation method of the target number of interpolated frames, or a preset calculation method.

[0124] For example, the electronic device can obtain the tail frame interpolation increment corresponding to the optical change point according to the mapping table, and add the tail frame interpolation increment corresponding to the optical change point to the initial tail frame interpolation number to obtain the target number of interpolated frames. The electronic device can also adjust the original tail zoom ratio interpolation method based on the tail frame interpolation increment corresponding to the optical change point to obtain the tail zoom ratio interpolation method corresponding to the optical change point; and then obtain the zoom ratio of the number of tail interpolated frames through the tail zoom ratio interpolation method corresponding to the optical change point; in this case, the tail frame interpolation increment corresponding to the optical change point is a parameter used to adjust the calculation method of the number of tail interpolated frames.

[0125] In this embodiment, when passing through the optical zoom point, the initial number of tail interpolation frames is increased by the tail interpolation increment to obtain the number of tail interpolation frames, so as to make the image smoothness better at the end of the zoom process.

[0126] In some embodiments, the difference between the zoom magnification at the end of the zoom process and the target zoom magnification is negatively correlated with the number of trailing frames. Therefore, if the difference at the end of the zoom process is too large, the number of trailing frames can be increased by methods such as differential or equal division to achieve a smooth exit effect at the end of the zoom process.

[0127] In some embodiments, the method further includes: when the current zoom level reaches the target zoom level, turning off all cameras except the camera belonging to the current zoom level.

[0128] The current zoom level reaching the target zoom level can be either the current zoom level and the target zoom level being equal, or the current zoom level and the target zoom level belonging to the same focal length.

[0129] In this embodiment, when the current zoom level reaches the target zoom level, all cameras and corresponding sensors except the camera belonging to the current zoom level are turned off to reduce power consumption.

[0130] In some embodiments, turning off all cameras except the camera belonging to the current zoom level includes: keeping each camera on for the duration of the zoom operation until the zoom operation ends, and then turning off all cameras except the camera belonging to the current zoom level.

[0131] In this embodiment, since zoom operations such as two-finger zoom and hardware zoom may update the target zoom ratio, the closing time of the secondary camera should be delayed accordingly based on the time the slider is held up, so as to obtain the image under dual cameras and thus obtain richer information.

[0132] In an exemplary embodiment, this solution is used for sliding zoom scenarios, i.e., sliding cut scenarios. In traditional technologies, sliding zoom solutions and user experiences lack clear strategies for handling the number of interpolated frames and the zoom curve used to represent the trend of field-of-view changes. Therefore, during sliding cuts, there may be abrupt and uneven smoothness issues with the field-of-view changes, or stuttering or retraction due to the secondary camera not being activated in time. Furthermore, for zoom reduction scenarios, due to hardware resource limitations, even with all three cameras active simultaneously for an extended period, there is still a need to switch cameras during fast sliding. During this process, one camera and its corresponding image sensor are turned off, and the freed-up resources are used to activate a new camera and its corresponding image sensor. Since turning a camera off and on generally requires approximately 7 frames, sufficient switching time must be ensured to prevent the field of view from changing to the corresponding range before the relevant camera has been activated, which would also cause problems. Currently, while there are frame interpolation solutions for sliding cut scenarios, they are only based on the angle of the zoom wheel; however, the difference in magnification and whether the optical zoom point has been crossed varies between different ranges. Therefore, for sliding cut scenarios, a more refined frame interpolation strategy is needed to balance power consumption, responsiveness, and the smoothness of field-of-view changes.

[0133] Based on this, this embodiment refines the scene and focal length using three dimensions: optical zoom point identification information, zoom speed information, and the direction of change in zoom magnification. This allows for the differentiated determination of the target zoom magnification for sliding zoom scenarios, thereby optimizing image processing during the zoom process. Priority is given to ensuring certain explicit needs and user experience, achieving a balance between responsiveness, smoothness, and power consumption.

[0134] The calculation of the initial number of interpolated frames and the initial number of end interpolated frames can be determined using the interpolation curve, i.e., the zoom curve. The zoom curve can be adjusted by the first N zoom ratios and the number of end interpolated frames, following the principle of rapid and uniform change in the intermediate state, and slow-in (fewer slow-in frames) and slow-out (more slow-out frames) to design the interpolation curve. The target number of interpolated frames and the number of end interpolated frames are determined by factors such as the optical zoom point marker information, zoom speed information, and the magnitude of the zoom ratio. For example, the optical zoom point marker information is used to determine whether to wait for the target lens to pull up and whether to insert more frames, thus comprehensively ensuring the overall zoom experience.

[0135] In an exemplary embodiment, when a user triggers a zoom operation, the zoom event sent by the camera application first determines whether it is a point-cut zoom scene or a sliding zoom scene. If it is a sliding zoom scene, the frame interpolation strategy described in this solution is followed, and operations 402-406 are executed to obtain the target number of interpolated frames. If it is a point-cut zoom scene, another solution is executed according to the point-cut strategy. When a sliding zoom scene is detected, further scene recognition and differentiation are required based on the zoom ratio of the current image frame, the zoom ratio of the previous frame, and the latest target zoom ratio, as well as a stop event flag. After determining the scene, the zoom ratio of the next frame is calculated according to the determined frame interpolation strategy, and the field of view is cropped according to the calculated zoom ratio value. Finally, when the user's zoom is detected to be finished, that is, when a stop event flag indicating that sliding will no longer occur is sent by the camera application, a fade-out process is performed at the end according to the tail curve, and the number of tail interpolated frames is calculated.

[0136] The zoom ratio of the current frame is used to determine whether the lens has reached the optical zoom point. The zoom ratio of the previous frame is used for error correction. The difference between the two latest target zoom ratios is compared with a threshold, and the zoom speed type is determined based on the interval in which the comparison result falls.

[0137] In one exemplary embodiment, as shown in FIG7, the embodiment in the sliding zoom scenario includes:

[0138] Operation 701: Open the camera and start image preview or recording.

[0139] Operation 702: The system detects that the user has adjusted the magnification (e.g., 0, 6X, 1X, 2X, 5X, 20X, etc.) or pulled up the magnification switching dial (slide), thereby triggering zoom.

[0140] Operation 703: Perform zoom mode identification to determine the zoom scene type.

[0141] Zoom mode recognition can obtain point-cut or sliding-cut scenes.

[0142] Operation 704: When it is a sliding zoom scene, the target number of interpolated frames is obtained based on the interpolation strategy that matches the current scene with the determined sliding zoom scene.

[0143] Operation 705, based on operation 704 and the equal division method or other methods, calculates the zoom ratio to determine the cropping parameters of the next frame, and crops the image to obtain the final field of view.

[0144] Operation 706: When a user zoom-off event is detected, update the interpolation curve to the tail interpolation curve, perform tail interpolation processing, and obtain the number of tail interpolated frames.

[0145] Operation 707, finally resume normal preview or recording. That is, after the current zoom level reaches the target zoom level, the secondary camera is turned off; the frame interpolation processing algorithm and the spatial alignment algorithm are turned off, completing the entire spatial alignment switching process; then preview or shooting processing continues.

[0146] In one embodiment, as shown in Figure 8, the included angle between any two pairs of dashed lines represents the opening and closing angles corresponding to the sliding cut. It can be noted that even with the same opening and closing angles in the high and low magnification ranges, the magnification change at the starting and ending points differs. In the low magnification range, even a small opening and closing angle can cause a significant change in the field of view. Therefore, it is unreasonable to interpolate frames solely based on the opening and closing angles for different focal lengths, as this will lead to inconsistent smoothness of the zoom curve. Thus, frame interpolation is necessary for a smooth transition.

[0147] In an exemplary embodiment, the frame interpolation strategy for the three dimensions of optical zoom point identification information, zoom speed information, and zoom magnification change is shown in Figure 9. The fast zoom type can be represented by the mask value; when zooming fast, all three cameras are pulled up, and when zooming slowly, the zoom can be taken gradually.

[0148] For zoom-in scenes, the field of view gradually decreases, so even if the secondary camera doesn't have time to widen its field of view, it's sufficient for cropping. For zoom-in scenes, the focus is on responsiveness and smoothness of zooming. Generally, the camera and corresponding image sensor can be activated before zooming stops. For slow transitions across optical zoom points in zoom-in and zoom-out scenes, multiple frames are interpolated before the optical zoom point. In this embodiment, the first interpolation increment can be 3, 4, or 5 frames (multiple zoom levels). This first interpolation increment provides a smoothing effect, thus providing more double frames for alignment and fusion animation.

[0149] For fast-slide scenes, the system first determines the target camera and its corresponding image sensor based on the latest zoom level. Then, it simultaneously activates all cameras and their corresponding image sensors that pass through the target frame, as well as the target camera and its corresponding image sensor. If these cameras and their corresponding image sensors are already active, their status remains unchanged. Simultaneously, when a new target zoom level is detected that crosses a light change point compared to the current frame's zoom level, the number of interpolated frames is increased accordingly. It's important to note that the added frames in this scenario must be before the light change point. In general, at least one additional interpolated frame is added before each light change point; that is, the second interpolated frame increment can be 1, 2, or 3 frames, etc., and is not limited to these.

[0150] For fast sliding and zoom reduction scenarios, firstly, based on the latest zoom ratio, the target camera and its corresponding image sensor are determined, and all intermediate cameras and their corresponding image sensors, as well as the target camera and its corresponding image sensor, are simultaneously activated. If the target's intermediate cameras and their corresponding image sensors, as well as the target camera and its corresponding image sensor, are already activated, their status remains unchanged. Simultaneously, when a new target zoom ratio is detected that crosses a light change point compared to the current frame's zoom ratio, the number of interpolated frames is increased accordingly. It's important to note that the added frames in this scenario must be before the light change point. In general, one additional interpolated frame needs to be added before each light change point; that is, the second interpolation increment is 1 frame.

[0151] Although both are fast-slide scenes, the frame interpolation strategies for fast slides at zoom magnification and zoom reduction need to differ, especially when the zoom level doesn't exceed the optical change point. This is mainly because while the judgment of fast slides is relatively accurate for zoom cuts, it's not accurate to determine whether the optical change point will be exceeded. For zoom magnification scenes, even if the judgment is delayed or misjudged, and even if the secondary camera doesn't activate in time, the field of view is sufficient to support cropping, so the impact is minimal. However, for zoom reduction scenes, the possibility that a fast slide not exceeding the optical change point refers to an intermediate state where the zoom level might still exceed the optical change point. Therefore, frame interpolation cannot simply be based on smoothness at the end. After determining the target number of interpolation frames for the zoom level, methods such as equal division can be used to determine the slope of the interpolation curve during rapid changes. This process continues until the app continuously sends two or three consistent zoom levels, indicating that the electronic device has entered the end stage of the zoom process, and then smooth frame interpolation at the end is performed. This solution allows us to determine whether a sliding scene passes through a light-changing point.

[0152] In an exemplary embodiment, the frame interpolation curve at the end of a fast-slide scene is shown in Figure 10. The initial frames of the interpolation curve are uniform. After the camera application issues a slide stop event, smooth frame interpolation is then applied to the zoom ratio. Therefore, this requires that the slide stop event also be issued promptly based on the Hardware Abstraction Layer (HAL) interface. Furthermore, the target zoom ratios in the figure are different, and the times at which the zoom process ends are different. In this case, the difference between the zoom ratio at the end of the zoom process and the target zoom ratio shows a negative correlation with the number of frame interpolations at the end.

[0153] In addition, for frame interpolation in sliding scenes, frame interpolation is not performed every time a new zoom ratio issued by the camera application is detected. In the case of the first zoom rate type, if the difference is less than 0.1, frame interpolation is no longer used to determine the zoom ratio. At this time, the change in the field of view is already slow enough, and the relationship of the magnification can be directly added, and the image is captured according to the magnification provided by the user.

[0154] Based on this, unlike the current spatial alignment and sliding cut schemes that only interpolate frames based on angles, this embodiment performs targeted and differentiated segmentation and scene-specific optimizations to address the complexity of sliding cut scenarios (random stopping points, random sliding speeds, and random sliding directions). First, it ensures the smoothness of stopping and the change in field of view for zoom magnification scenes and zoom reduction scenes that do not pass the optical change point. Second, for slow sliding scenes, it also uses multiple frame interpolation before the optical change point to provide more double frames for alignment and motion effects, resulting in better alignment and dynamic effects. Finally, for the complex sliding cut scenario of zoom reduction quickly passing the optical change point, it uses uniform multiple frame interpolation before the optical change point to give the secondary camera time to activate, avoiding excessive power consumption due to the secondary camera pulling up the focal length for too long, thus achieving a balance between smoothness and power consumption.

[0155] In this context, by inserting virtual transition frames during spatial alignment switching based on the characteristics of scene changes, the switching effect is made smoother, improving the user experience. Specifically, by subdividing and organizing the sliding scenes, differentiating between zoom magnification, zoom reduction, fast sliding, slow sliding, and transitions to optical zoom points, a differentiated frame interpolation strategy is implemented to achieve a balance between smoothness and responsiveness. Different frame interpolation strategies are applied to different sliding scenes by considering the responsiveness of stopping and the smoothness of changes in the field of view, while also taking into account curve adjustments.

[0156] For example, by comparing the zoom ratio of the current frame with that of the previous frame, and combining the changes in the target zoom ratio with the stop-swipe event issued by the app, different sliding scenarios (zoom in, zoom out, fast slide, slow slide, over optical zoom point, etc.) are comprehensively judged as the basis for subsequent frame interpolation and curve adjustment. Meanwhile, the differentiated frame interpolation strategy, i.e., the zoom ratio mapping strategy, needs to be adjusted with emphasis for different scenarios. For example, for ordinary zoom in scenarios, since there is no issue of insufficient field of view, the focus can be on the responsiveness of stopping and the smoothness of the field of view change. However, for zoom out scenarios, especially fast-slide zoom out over optical zoom points, the number of interpolated frames needs to be sufficient to allow time for the secondary camera to activate, avoiding insufficient field of view cropping due to the secondary camera not activating in time, thus preventing image jitter or pauses.

[0157] It should be understood that although the operations in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these operations are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these operations, and they can be executed in other orders. Moreover, at least some of the operations in the flowcharts of the embodiments described above may include multiple operations or multiple stages. These operations or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these operations or stages is not necessarily sequential, but can be performed alternately or in turn with other operations or at least some of the operations or stages in other operations.

[0158] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0159] In an exemplary embodiment, as shown in FIG11, an image processing apparatus is provided, comprising:

[0160] The acquisition module 1102 is used to acquire the current zoom ratio and the target zoom ratio issued by the camera application corresponding to the current image frame in response to the zoom operation; wherein, the current zoom ratio is used to determine the optical zoom point identification information, and the target zoom ratio is used to determine the zoom speed information.

[0161] Scene module 1104 is used to determine the number of target frames based on the optical change point identification information and the zoom speed information;

[0162] The processing module 1106 is used to process the target image using the zoom ratio of the target interpolation frame number.

[0163] In one embodiment, the scene module 1104 is used for:

[0164] When the optical zoom point identification information indicates that the zoom process does not pass through the optical zoom point, the target number of interpolated frames is determined according to the number of interpolated frames corresponding to the zoom speed information.

[0165] When the optical zoom point identification information indicates that the zoom process passes through the optical zoom point, the interpolation increment corresponding to the optical zoom point is obtained, and the target number of interpolation frames is obtained based on the interpolation increment and the number of interpolation frames corresponding to the zoom speed information.

[0166] In one embodiment, the scene module 1104 is used for:

[0167] When the zoom speed information represents a first zoom speed type, a first initial number of interpolated frames is obtained, and based on the first initial number of interpolated frames, the number of interpolated frames corresponding to the zoom speed information is obtained.

[0168] When the zoom speed information represents a second zoom speed type, a second initial number of interpolated frames is obtained, and based on the second initial number of interpolated frames, the number of interpolated frames corresponding to the zoom speed information is obtained.

[0169] Wherein, the first initial number of interpolated frames is less than the second initial number of interpolated frames; the zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type.

[0170] In one embodiment, in the zoom ratio sequence corresponding to the second initial number of interpolated frames, the difference between adjacent zoom ratios in the first N zoom ratios is less than the difference between adjacent zoom ratios in the zoom ratios after the Nth zoom ratio.

[0171] In one embodiment, the processing module 1106 is configured to:

[0172] When the zoom speed information represents a second zoom speed type, the target optical zoom point is determined based on the current zoom magnification and the target zoom magnification.

[0173] The camera to be switched is determined based on the target light change point;

[0174] The camera to be switched on is controlled to be in the on state.

[0175] In one embodiment, the scene module 1104 is used for:

[0176] When the zoom speed information represents the first zoom speed type, the first interpolation increment corresponding to the optical zoom point is obtained;

[0177] When the zoom speed information represents the second zoom speed type, the second interpolation increment corresponding to the optical zoom point is obtained;

[0178] The number of interpolated frames increased by the first interpolation increment is greater than the number of interpolated frames increased by the second interpolation increment.

[0179] In one embodiment, the acquisition module 1102 is configured to:

[0180] When there is an optical zoom point between the current zoom ratio and the target zoom ratio, optical zoom point identification information is obtained to indicate that the zoom process passes through the optical zoom point;

[0181] If there is no optical zoom point between the current zoom ratio and the target zoom ratio, optical zoom point identification information is obtained to indicate that the zoom process does not pass through an optical zoom point.

[0182] In one embodiment, the acquisition module 1102 is configured to:

[0183] The difference between different target zoom ratios is compared with a difference threshold.

[0184] When the difference value is less than the difference threshold, the zoom speed information is determined to be the first zoom speed type;

[0185] When the difference value is greater than or equal to the difference threshold, the zoom speed information is determined to be a second zoom speed type.

[0186] In one embodiment, the acquisition module 1102 is configured to:

[0187] If the current zoom ratio is greater than the target zoom ratio, the optical zoom point identification information is determined based on the current zoom ratio.

[0188] In one embodiment, the processing module 1106 is configured to:

[0189] If the current zoom ratio is less than the target zoom ratio, obtain the initial number of interpolated frames;

[0190] The image is processed using the zoom magnification of the initial number of interpolated frames.

[0191] In one embodiment, the initial number of interpolated frames includes a first initial number of interpolated frames and a second initial number of interpolated frames; the first initial number of interpolated frames is the number of interpolated frames under the first zoom speed type, and the second initial number of interpolated frames is the number of interpolated frames under the second zoom speed type.

[0192] In one embodiment, after processing the target image using the zoom ratio of the target interpolation frame count, the acquisition module 1102 is configured to:

[0193] When the target zoom magnification is continuously and matched to a preset number, the number of tail frames corresponding to the end stage of the zoom process is determined.

[0194] The tail image is processed using the zoom ratio of the number of tail interpolated frames; the tail image is acquired later than the target image.

[0195] In one embodiment, after processing the target image using the zoom ratio of the target interpolation frame count, the acquisition module 1102 is configured to:

[0196] If a stop event flag is detected, determine the number of trailing frames corresponding to the end phase of the zoom process;

[0197] The tail image is processed using the zoom ratio of the number of tail interpolated frames; the tail image is acquired later than the target image.

[0198] In one embodiment, when the optical zoom point identification information indicates that the zoom process passes through an optical zoom point, the acquisition module 1102 is used to:

[0199] Obtain the initial number of tail interpolated frames and the tail interpolated frame increment corresponding to the optical change point. Based on the tail interpolated frame increment corresponding to the optical change point and the initial number of tail interpolated frames, obtain the number of tail interpolated frames.

[0200] In one embodiment, the processing module 1106 is configured to:

[0201] When the current zoom level reaches the target zoom level, all cameras except the camera associated with the current zoom level are turned off.

[0202] In one embodiment, the processing module 1106 is configured to:

[0203] Each camera remains on for the duration of the zoom operation until the zoom operation ends, at which point all cameras except the camera corresponding to the current zoom level are turned off.

[0204] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0205] In an exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 12. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the electronic device provides computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the electronic device is used for exchanging information between the processor and external devices. The communication interface of the electronic device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the electronic device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0206] Those skilled in the art will understand that the structure shown in Figure 12 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0207] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the operations described in the above method embodiments.

[0208] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, performs the operations described in the above method embodiments.

[0209] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the operations described in the above method embodiments.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0211] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0212] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0213] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized by, The method includes: In response to a zoom operation, the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application are obtained, wherein the current zoom ratio is used to determine the optical zoom point identification information, and the target zoom ratio is used to determine the zoom speed information. The number of target frames is determined based on the optical zoom point identification information and the zoom speed information; and The target image is processed using the zoom ratio of the target number of interpolated frames.

2. The method of claim 1, wherein, Determining the target frame interpolation number based on the optical change point identification information and the zoom speed information includes: When the optical zoom point identification information indicates that the zoom process does not pass through an optical zoom point, the target number of interpolated frames is determined based on the number of interpolated frames corresponding to the zoom speed information; and When the optical zoom point identification information indicates that the zoom process passes through the optical zoom point, the interpolation increment corresponding to the optical zoom point is obtained, and the target number of interpolation frames is obtained based on the interpolation increment and the number of interpolation frames corresponding to the zoom speed information.

3. The method of claim 2, wherein, The method for determining the number of interpolated frames corresponding to the zoom speed information includes: When the zoom speed information represents a first zoom speed type, a first initial number of interpolated frames is obtained, and based on the first initial number of interpolated frames, the number of interpolated frames corresponding to the zoom speed information is obtained; and When the zoom speed information represents a second zoom speed type, a second initial number of interpolated frames is obtained, and based on the second initial number of interpolated frames, the number of interpolated frames corresponding to the zoom speed information is obtained. Wherein, the first initial number of interpolated frames is less than the second initial number of interpolated frames; the zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type.

4. The method of claim 3, wherein, In the zoom ratio sequence corresponding to the second initial number of interpolated frames, the difference between adjacent zoom ratios in the first N zoom ratios is less than the difference between adjacent zoom ratios in the zoom ratios after the Nth zoom ratio.

5. The method of claim 3, wherein, The method further includes: When the zoom speed information represents a second zoom speed type, the target optical zoom point is determined based on the current zoom magnification and the target zoom magnification. The camera to be switched is determined based on the target light change point; and The camera to be switched on is controlled to be in the on state.

6. The method of claim 3, wherein, The step of obtaining the interpolation increment corresponding to the optical change point includes: When the zoom speed information represents the first zoom speed type, obtain the first frame interpolation increment corresponding to the optical change point; and When the zoom speed information represents the second zoom speed type, the second interpolation increment corresponding to the optical zoom point is obtained; The number of interpolated frames increased by the first interpolation increment is greater than the number of interpolated frames increased by the second interpolation increment.

7. The method of claim 1, wherein, The method for determining the optical zoom point identification information using the current zoom magnification includes: When an optical zoom point exists between the current zoom level and the target zoom level, optical zoom point identification information is obtained to indicate that the zoom process has passed through the optical zoom point; and If there is no optical zoom point between the current zoom ratio and the target zoom ratio, optical zoom point identification information is obtained to indicate that the zoom process does not pass through an optical zoom point.

8. The method of claim 1, wherein, The method by which the target zoom ratio is used to determine zoom speed information includes: The difference between different target zoom ratios is compared with a difference threshold. When the difference value is less than the difference threshold, the zoom speed information is determined to be a first zoom speed type; and When the difference value is greater than or equal to the difference threshold, the zoom speed information is determined to be a second zoom speed type.

9. The method of claim 1, wherein, The method for determining the optical zoom point identification information using the current zoom magnification includes: If the current zoom ratio is greater than the target zoom ratio, the optical zoom point identification information is determined based on the current zoom ratio.

10. The method of claim 9, wherein, The method further includes: When the current zoom ratio is less than the target zoom ratio, the initial number of interpolated frames is obtained; and The image is processed using the zoom magnification of the initial number of interpolated frames.

11. The method of claim 10, wherein, The initial frame interpolation number includes a first initial frame interpolation number and a second initial frame interpolation number; the first initial frame interpolation number is the number of frames interpolated under the first zoom speed type, and the second initial frame interpolation number is the number of frames interpolated under the second zoom speed type.

12. The method according to claim 1, characterized in that, After processing the target image using the zoom ratio of the target interpolation frame number, the method further includes: When the target zoom magnification is continuously and consistently reached in a predetermined manner, the number of trailing frames corresponding to the end phase of the zoom process is determined; and The tail image is processed using the zoom ratio of the number of tail interpolated frames; the tail image is acquired later than the target image.

13. The method according to claim 1, characterized in that, After processing the target image using the zoom ratio of the target interpolation frame number, the method further includes: Upon detecting a stop event flag, determine the number of trailing frames corresponding to the end phase of the zoom process; and The tail image is processed using the zoom ratio of the number of tail interpolated frames; the tail image is acquired later than the target image.

14. The method according to claim 13, characterized in that, When the optical zoom point identification information indicates that the zoom process has passed through an optical zoom point, determining the number of trailing frames corresponding to the end stage of the zoom process includes: Obtain the initial number of tail interpolated frames and the tail interpolated frame increment corresponding to the optical change point. Based on the tail interpolated frame increment corresponding to the optical change point and the initial number of tail interpolated frames, obtain the number of tail interpolated frames.

15. The method according to claim 1, characterized in that, The method further includes: When the current zoom level reaches the target zoom level, all cameras except the camera associated with the current zoom level are turned off.

16. The method according to claim 14, characterized in that, The step of turning off all cameras except the camera corresponding to the current zoom level includes: Each camera remains on for the duration of the zoom operation until the zoom operation ends, at which point all cameras except the camera corresponding to the current zoom level are turned off.

17. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the current zoom ratio and the target zoom ratio issued by the camera application in response to the zoom operation; wherein, the current zoom ratio is used to determine the optical zoom point identification information, and the target zoom ratio is used to determine the zoom speed information. The scene module is used to determine the number of target frames to be interpolated based on the optical change point identification information and the zoom speed information; and The processing module is used to process the target image using the zoom ratio of the target interpolation frame number.

18. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it performs the operation of the method according to any one of claims 1 to 16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the operation of the method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it performs the operation of the method according to any one of claims 1 to 16.

Images