Image processing methods and apparatuses, and related products

Abstract

The present disclosure provides an image processing method and apparatus, and related products. The method includes: obtaining first angular data of M frames and second angular data of N frames prior to the M frames, where M and N are positive integers; determining, based on the first angular data and the second angular data, one or more motion parameters of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured; and processing, based on the one or more motion parameters, the M frames using an image stabilization strategy corresponding to the one or more motion parameters.

Description

[0001] IMAGE PROCESSING METHODS AND APPARATUSES, AND RELATED PRODUCTS TECHNICAL FIELD

[0002] This disclosure relates generally to the field of image processing, and in particular to image processing methods, image processing apparatuses, electronic devices, and related products. BACKGROUND

[0003] Unintended motions that occur when holding a camera or a smartphone equipped with the camera to capture images or videos, especially during handheld shooting scenarios, may cause blur in frames captured by the camera, such as shakes or jitters from hands. Image stabilization (IS) is widely used in photography and videography to mitigate the blur caused by the unintended camera motions, transforming original shaky frames into clearer and more stable frames. However, image stabilization still has poor performance in many shooting scenarios.

[0004] SUMMARY

[0005] The present disclosure provides image processing methods, image processing apparatuses, electronic devices, and related products, to stabilize blurred frames caused by unintended camera motions and improve stabilization performance.

[0006] According to a first aspect, an image processing method is described. The method may be performed by an electronic device (e.g. a smartphone), or the method may be performed by a module, a circuit, a unit, or a chip in the electronic device where the module, the circuit, the unit, or the chip is responsible for the image processing method in the electronic device. The method includes: obtaining first angular data of M frames and second angular data of N frames prior to the M frames, where the first angular data includes one or more rotation angles of a camera when capturing the M frames, the second angular data includes one or more rotation angles of the camera when capturing the N frames, and M and N are positive integers; determining, based on the first angular data and the second angular data, one or more motion parameters of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured; and processing, based on the one or more motion parameters, the M frames using an image stabilization strategy corresponding to the one or more motion parameters.

[0007] According to this method, the first angular data of the M frames and the second angular data of the N frames are obtained, where the N frames are prior to the M frames; the one or more motion parameters of the camera during the period from the time at which the first frame of the N frames is captured to the time at which the last frame of the M frames is captured are determined based on the first angular data and the second angular data; and the M frames is processed using the image stabilization strategy corresponding to the one or more motion parameters. The one or more motion parameters during the period from the time at which the first frame of the N frames is captured to the time at which the last frame of the M frames is captured are a more precise camera motion characterization that aligns with the current capturing scenario, the M frames may be processed accurately by the image stabilization strategy corresponding to the one or more motion parameters, and the stabilization performance on the M frames may be improved. Therefore, using this method may have more optimal stabilization performance. Furthermore, since the image stabilization strategy for processing the frames may change with the one or more motion parameters of the frames, this method may provide the image stabilization performance well in a variety of scenarios.

[0008] In some examples, the one or more motion parameters include at least one of: the first angular data and the second angular data, an angular velocity, an angular acceleration, a standard deviation of angle, a kurtosis of velocity, a max distance of upper peaks, a skewness of angle, a velocity peak, a kurtosis of acceleration, a standard deviation of velocity or an acceleration peak of the camera during the period, a recent velocity value or a sine of angle of the camera when capturing the last frame in the M frames.

[0009] According to the examples, the one or more motion parameters includes various kinds of parameters for describing motions of the camera capturing frames from different perspectives. Therefore, the one or more motion parameters may provide a more precise camera motion characterization, which may help the determination of the image stabilization to process frames.

[0010] In some examples, processing, based on the one or more motion parameters, the M frames using the image stabilization strategy corresponding to the one or more motion parameters, includes: determining, based the one or more motion parameters, whether a target motion type of the M frames is determined; and processing the M frames using the image stabilization strategy corresponding to the target motion type in a case where the target motion type is determined.

[0011] According to the examples, it is determined whether the target motion type of the M frames is determined, and in a case where the target motion type is determined, the image stabilization strategy corresponding to the target motion type may be used to process the M frames. Because patterns of blurs may be similar for frames within the same motion type, a corresponding image stabilization strategy applied on frames within the same motion type may provide a simpler and more generalized way to process the M frames.

[0012] In some examples, the method further includes processing the M frames using a default image stabilization strategy in a case where the target motion type is not determined.

[0013] According to the examples, the default image stabilization strategy may be used to process frames in a case where the target motion type is not determined. Therefore, for one or more frames that are shot by the camera with motions that are difficult to classify, compared to determining a specific motion type that is not well-suited for the current capturing scenario and determining a specific image stabilization strategy that may not be suitable to the current capturing scenario, using the default image stabilization strategy for processing frames may have better stabilization performance.

[0014] In some examples, determining whether the target motion type of the M frames is determined includes: obtaining an output of a classification model by inputting the one or more motion parameters into the classification model, where the classification model is trained based on a plurality of sample motion parameters and a plurality of motion types of the plurality of sample motion parameters; determining, based on the output, whether the target motion type of the M frames is determined.

[0015] According to the examples, the one or more motion parameters are inputted into the classification model, and it is determined whether the target motion type of the M frames is determined, based on the output of the classification model. For example, the classification model outputs the target motion type directly or the classification model outputs probability information and the target motion type may be determined based on the probability information. Using the output of the classification model to determine whether the target motion type of the M frames is determined, this method may have better robustness.

[0016] In some examples, the output includes a plurality of probabilities of the plurality of motion types; and determining, based on the output, whether the target motion type of the M frames is determined includes: determining a first motion type of the plurality of motion types as the target motion type in a case where a probability of the first motion type is greater than a first threshold.

[0017] According to the examples, the output includes the plurality of probabilities of the plurality of motion types; and the first motion type of the plurality of motion types is determined as the target motion type in a case where the probability of the first motion type is greater than the first threshold. The probabilities may indicate the certainty degrees of motion type classification results, where the probability is higher, and the certainty degree is higher, so that the first motion type having the probability greater than the first threshold may be determined as the target motion type, and the image stabilization strategy corresponding to the target motion type may be more suitable to process frames.

[0018] In some examples, determining, based on the output, whether the target motion type of the M frames is determined further includes: determining that the target motion type is not determined, in a case where the plurality of probabilities are all less than or equal to the first threshold.

[0019] According to the examples, in a case where there is no probability that is greater than the first threshold, which may indicate a high uncertainty of the classification result, and all the motion types may not be target suitable for frames, so that target motion type may not be determined. Compared with determining the target motion type regardless of whether there is a probability greater than the first threshold, not determining the target type may help avoid inaccurate determination of the target motion type and processing the frames using the subsequent inaccurate image stabilization strategy.

[0020] In some examples, the target motion type is a hand still motion, a huge directed motion, or a high-frequency motion; and in the hand still motion, an angular value of the M frames is less than a second threshold, and a jitter frequency of the M frames is less than a third threshold; in the huge directed motion, the angular value of the M frames is greater than or equal to the second threshold, and the jitter frequency of the M frames is less than the third threshold; and in the high-frequency motion, the angular value of the M frames is less than the second threshold, and the jitter frequency of the M frames is greater than or equal to the third threshold.

[0021] According to the examples, three motion types including hand still motion, huge directed motion, and high-frequency motion are described. These three motion types may occur frequently in different camera shooting scenarios, therefore an image processing method that uses different image stabilization strategies corresponding to these three motion types may be applied across a wide range of scenarios.

[0022] In some examples, N is greater than or equal to a fourth threshold.

[0023] According to the examples, where N is greater than or equal to the fourth threshold, the usage of the fourth threshold provides a condition to determine whether there are sufficient frames prior to the M frames. There are sufficient frames prior to the M frames (that is, N is greater than or equal to the fourth threshold), so that the M frames may be processed by the image stabilization strategy corresponding to the one or more motion parameters that are obtained according to the M frames and N frames. Therefore, the M frames may be processed by a more suitable image stabilization strategy.

[0024] According to a second aspect, an image processing apparatus is described. The apparatus may be configured in an electronic device (e.g. a smartphone), or the apparatus may be configured as a module, a circuit, a unit, or a chip in the electronic device where the module, the circuit, the unit, or the chip is responsible for the image processing. The apparatus includes various units for performing the method according to the first aspect or any example in the first aspect. The apparatus includes: an obtaining unit and a processing unit. The obtaining unit may be configured to obtain first angular data of M frames and second angular data of N frames prior to the M frames, where the first angular data includes one or more rotation angles of a camera when capturing the M frames, the second angular data includes one or more rotation angles of the camera when capturing the N frames, and M and N are positive integers. The processing unit may be configured to determine, based on the first angular data and the second angular data, one or more motion parameters of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured; and process, based on the one or more motion parameters, the M frames using an image stabilization strategy corresponding to the one or more motion parameters.

[0025] In some examples, the processing unit is configured to determine, based the one or more motion parameters, whether a target motion type of the M frames is determined; and process the M frames using the image stabilization strategy corresponding to the target motion type in a case where the target motion type is determined.

[0026] In some examples, the processing unit is configured to process the M frames using a default image stabilization strategy in a case where the target motion type is not determined.

[0027] In some examples, the processing unit is configured to obtain an output of a classification model by inputting the one or more motion parameters into the classification model, where the classification model is trained based on a plurality of sample motion parameters and a plurality of motion types of the plurality of sample motion parameters; and determine, based on the output, whether the target motion type of the M frames is determined.

[0028] In some examples, the output includes a plurality of probabilities of the plurality of motion types; and the processing unit is configured to determine a first motion type of the plurality of motion types as the target motion type in a case where a probability of the first motion type is greater than a first threshold.

[0029] In some examples, the processing unit is configured to determine that the target motion type is not determined, in a case where the plurality of probabilities are all less than or equal to the first threshold.

[0030] According to a third aspect, an electronic device is described. The electronic device includes processing circuitry for performing the method according to the first aspect or any example in the first aspect.

[0031] According to a fourth aspect, a chip is provided. The chip includes an input / output (I / O) interface and a processor, where the processor is configured to call and run a computer program stored in a memory, to enable a device installing with the chip to execute the method according to the first aspect or any example in the first aspect.

[0032] According to a fifth aspect, an electronic device is described. The electronic device includes one or more processors; and a memory coupled to the one or more processors and storing instructions for execution by the processors, where the instructions, when executed by the one or more processors, cause the device to perform the method according to the first aspect or any example in the first aspect. In some examples, the electronic device may further include an interface circuit, and the processor is configured to communicate with another apparatus / device or component through the interface circuit.

[0033] According to a sixth aspect, a computer-readable storage medium is described. The computer-readable storage medium stores computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer is enabled to perform the method according to the first aspect or any example in the first aspect.

[0034] According to a seventh aspect, a computer program product is described. When a computer reads and executes the computer program product, the computer is enabled to perform the method according to the first aspect or any example in the first aspect.

[0035] The present disclosure encompasses various embodiments, including not only method embodiments, but also other embodiments such as apparatus / device embodiments and embodiments related to non-transitory computer readable storage media. Embodiments may incorporate, individually or in combinations, the features disclosed herein.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Numerous details are described herein to provide a thorough understanding of embodiments illustrated in accompanying drawings. However, some embodiments may be practiced without many of the specific details, and the scope of the claims is only limited by those features and aspects specifically recited in the claims. Furthermore, well-known processes, components, and materials have not necessarily been described in exhaustive detail to avoid obscuring pertinent aspects of the embodiments described herein.

[0038] FIG. 1 is a schematic diagram of a process of obtaining stabilized frames from shaky frames in accordance with some embodiments of the present disclosure;

[0039] FIG. 2 is a schematic diagram of an electronic image stabilization process in accordance with some embodiments of the present disclosure;

[0040] FIG. 3 is a schematic diagram of a system to which some embodiments of the present disclosure are applicable;

[0041] FIG. 4 is a block diagram of functional units of a processor in FIG. 1 that implements an image processing method, in accordance with some embodiments of the present disclosure;

[0042] FIG. 5 is a schematic flowchart of an image processing method in accordance with some embodiments of the present disclosure;

[0043] FIG. 6 is a schematic flowchart of a process for extracting features in accordance with some embodiments of the present disclosure; FIG. 7 is a schematic flowchart of a process for determining an image stabilization strategy in accordance with some embodiments of the present disclosure;

[0044] FIG. 8 is a schematic diagram of visualizations of three motion types in accordance with some embodiments of the present disclosure;

[0045] FIG. 9 is a block diagram of an image processing apparatus in accordance with some embodiments of the present disclosure; and

[0046] FIG. 10 is a block diagram of an electronic device in accordance with some embodiments of the present disclosure.

[0047] DETAILED DESCRIPTION

[0048] In the following description, reference is made to the accompanying drawings, which form part of the present disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and include structural or logical changes not depicted in the drawings. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0049] The embodiments set forth herein represent information sufficient to practice the claimed subject matter and illustrate ways of practicing such subject matter. Upon reading the following description in light of the accompanying figures, those skilled in the art will understand the concepts of the claimed subject matter and will recognize applications of these concepts not particularly addressed herein. It is understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.

[0050] Cameras or smartphones equipped with cameras are widely used by users to capture images or videos in their daily lives. When users hold cameras or smartphones to capture images or videos, especially during handheld shooting situations, unintended camera motions may occur such as shakes or jitters from hands, which may cause blur in frames captured by the camera. Image stabilization (IS) is widely used in photography and videography to mitigate the blur in frames caused by these unintended camera motions, transforming original shaky frames into clearer and more stable frames. That is, the image stabilization refers to manners used to reduce blur in images or videos.

[0051] Referring to FIG. 1, as an illustrative example without limitation, a simplified schematic illustration of obtaining stable frames from shaky frames in accordance with some embodiments of the present disclosure. As shown in FIG. 1, Rt, Rt+1, and Rt+2 represent shaky frames at moments t, t+1, and t+2, and blank dots in shaky frames Rt, Rt+1, and Rt+2 represent center points of the shaky frames Rt, Rt+1, and Rt+2. St, St+1, and St+2 represent stabilized frames at the moments of time t, t+1, and t+2, and black dots in the stabilized frames St, St+1, and St+2 represent center points of the stabilized frames St, St+1, and St+2. In addition, the dashed lines represent a transformation from the shaky frames Rt, Rt+1, and Rt+2 to the stabilized frames St, St+1 and St+2. As shown in FIG. 1, the stabilized frames are less distorted than the corresponding shaky frames. The process of obtaining stabilized frames from shaky frames may refer to image stabilization problem

[0052] The above image stabilization problem may be mathematically formulated as follows:

[0053] St=Ft·Rt, where Rt denotes the shaky frames captured by camera, Ft is a transformation applied to Rt, St is the stabilized frames obtained by applying Ft on Rt. All types of image transformations may be considered for a correction. The widely-used corrections are rotation vectors (3 degree of freedom (3 DOF)), rotation matrices (3 DOF), affine transform matrices (6 DOF), and homography matrices (8DOF). The degree of freedom refers to the number of dimensions in which the image can be adjusted or transformed, for example, for rotation vectors with 3 DOF, the dimensions in which the image can be adjusted or transformed may be three types of angles.

[0054] There are two types of image stabilization: optical image stabilization (OIS) and electronic image stabilization (EIS). The OIS may counteract the unintended camera motions by moving a lens to cancel the unintended camera motions. The EIS is a software-based approach that compensates for unintended camera motions by cropping frames and adjusting the frames in postprocessing. The EIS is useful in action cameras and smartphones, where mechanical stabilization systems may be impractical. Therefore, the EIS may stabilize frames in dynamic shooting scenarios.

[0055] FIG. 2 illustrates a schematic diagram of a process of an electronic image stabilization method. As shown in FIG. 2, the EIS may start from receiving original frames, and include the following processes: camera motion estimation, motion compensation, and image warping, and stabilized frames are obtained.

[0056] Motion estimation is a process of determining the motion of the camera between adjacent frames. The motion of the camera may be a rotation motion and / or translation motion of the camera. The camera motion estimation may be done by integrating motion data or by feature matching. The raw motion data may be obtained from the motion sensors and used to determine the rotation motion and / or translation motion of the camera. Alternatively, the feature matching between adjacent frames may be performed to determine the rotation motion and / or translation motion of the camera, for example, certain points (feature points) in the pixel space of an frame may be computed and corresponding points in another image (adjacent to the computed image) may be found, and the certain points and corresponding points may be used to be computed to obtain the rotation and / or transformation between the two adjacent images as the rotation motion and / or translation motion of the camera.

[0057] Motion compensation is a process that addresses a challenge posed by the camera motion during shooting and is responsible for correcting the original frame trajectory, complying with restrictions imposed on smartphones. The original frame trajectory may include a sequence of rotation motions and / or translation motions.

[0058] Image warping is a process that geometrically distorts a frame to transform the frame from one perspective to another perspective. In image warping, original locations of the pixels of the frame may be mapped to new locations according to the corrected frame trajectory. Therefore, the frame may be transformed to a new, stabilized frame.

[0059] Although the EIS may stabilize frames in dynamic shooting scenarios, users face different camera dynamic scenarios on a daily basis and different camera dynamic scenarios have various user expectations on stabilization behaviors. For example, in scenarios with high camera tremors, such as running scenarios or walking scenarios, a user expects an excessive smoothing experience to maintain a video captured as stable as possible, comparable with a cinematic camera. For another example, there are dynamic scenarios, such as the user follows an animal with the camera or a playing child in the playground to record memories (panning) - in these types of scenarios, the user expects frames captured will ‘follow the hand’ of the user.

[0060] In view of this, this disclosure provides an image processing method. For frames captured in a scenario, one or more motion parameters are determined to characterize motions of the camera that captures the frames. The frames may be processed by an image stabilization strategy corresponding to the one or more motion parameters. The image stabilization strategy corresponding to the one or more motion parameters may align with the current capturing scenario and the stabilization performance may be improved. Since the image stabilization strategy for processing the frames may change with the one or more motion parameters of the frames, the method provided in the present disclosure may provide the image stabilization performance well in a variety of dynamic scenarios.

[0061] It is noted that the present disclosure focuses on the EIS, and the term “EIS” may also refer to “IS” for concision below. It is also noted that the term “camera” may also refer to “camera or action camera or smartphone” for concision below.

[0062] To better understand some embodiments of the present disclosure, a system 300 applicable to some embodiments of the present disclosure is first introduced in combination with FIG. 3. As shown in FIG. 3, the system 300 includes an input unit 310, a processor 320, a storage unit 330, and a display unit 340.

[0063] In some embodiments, the system 300 may be configured on a single electronic device, such as, but not limited to, a smartphone or computing device with an integrated camera, an action camera, a camera, a wearable device with an action camera, or other hardware with an integrated camera. In these embodiments, the input unit 310, the processor 320, the storage unit 330, and the display unit 340 may communicate with each other through a bus, e.g. a peripheral component interconnect (PCI) bus, universal serial bus, institute of electrical and electronics engineers (IEEE) 1394 “Firewire” bus, small computer system interface (SCSI) bus, serial advanced technology attachment (Serial- ATA) bus, aeronautical radio incorporated (ARINC) bus, etc.), to which various units are electronically coupled.

[0064] In some other embodiments, the system 300 may be configured on a plurality of electronic devices. For example, the input unit 310 may be configured on a camera or an action camera, and the processor 320, the storage unit 330, and the display unit 340 may be configured on a smartphone, a computing device, a wearable device, or other hardware. The processor 320, the storage unit 330, and the display unit 340 may communicate with each other through the bus, as described above. Furthermore, the input unit 310 may communicate with any of the processor 320, the storage unit 330, and the display unit 340 through wired or wireless connections. For example, the wireless connection may be a cellular network, a wireless local area network (e.g. wireless fidelity (Wi-Fi™)), or the like. In addition, the processor 320, the storage unit 330, and the display unit 340 may be located in a cloud platform.

[0065] The input unit 310 may be configured to obtain two types of data: frames and raw motion data. The raw motion data may include one or more angle sets, where each angle set may include at least one of a rotation angle along a pitch axe, a rotation angle along a yaw axe, or a rotation angle along a roll axe. The rotation angle along the pitch axe, the rotation angle along the yaw axe, and the rotation angle along the roll axes may be obtained at the same time point or different time points during shooting, where the difference among different time points may be in the preset range. It is noted the pitch axe, yaw axe, and roll axe may be relative to the device where the input unit 310 is located.

[0066] In some embodiments, the input unit 310 may include an image-capturing device to obtain frames. The image-capturing device may be a digital camera, a webcam, a dashcam, or any other image-capturing device capable of generating frames.

[0067] In some other embodiments, the input unit 310 may include a motion-recording device to obtain raw motion data. The motion-recording device may be a gyroscope, a magnetometer, an accelerometer, an inertial measurement unit (IMU), or the like, where the IMU may be understood as an integration of the accelerometer, the gyroscope, the magnetometer, or other motion sensors.

[0068] io In still some other embodiments, the input unit 310 may include the above image-capturing device and motion-recording device. It is noted that the image-capturing device and motionrecording device may be integrated or separately.

[0069] The processor 320 may be a central processing unit (CPU), graphics processing unit (GPU), or other devices that are capable of executing instructions and performing computations.

[0070] The storage unit 330 may be configured to store program instructions suitable for being executed by the processor 310. Also, the storage unit 330 may be configured to store data from the input unit 310 and processor 320. The storage unit 330 may be configured in a form of a computer-readable storage medium, such as a volatile memory, random access memory (RAM), and / or a non-volatile memory, for example, flash memory.

[0071] The computer-readable storage medium may include a volatile or non-volatile type, removable or non-removable medium that may store information using any manner or strategy. Information may include computer-readable instructions, a data structure, a program module, or other data.

[0072] The display unit 340 may be configured to provide the processed frames to users. The display unit 340 may be configured by a screen such as a liquid crystal display (LCD), an organic light-emitting display (OLED), or other devices that are capable of displaying frames.

[0073] The above system 300 may implement an image processing method according to one or more embodiments of the present disclosure. For implementing the image processing method, a more detailed architecture of the processor 320 in FIG. 1 may be shown in FIG. 4. As shown in FIG. 4, the processor 320 includes a preprocessing unit 410, an analysis unit 420, and a processing unit 430. The analysis unit 420 includes a feature extraction unit 421 and a motion classification 422 for different analysis purposes, which will be introduced in detail in the following embodiments. The processing unit 430 includes a selection unit 431, an image stabilization unit 432, and an image warp unit 433 for different processing purposes, which will be introduced in detail in the following embodiments. It is noted that the above units in the processor 320 represent logical divisions of functionality rather than physical components. Therefore, if the processor 320 includes one or more processors, the above units may be located on different processors or on the same processor.

[0074] An image processing method 500 is depicted in FIG. 5, where the image processing method 500 may be implemented by the processor 320. In the following, the image processing method 500 is described with reference to FIG. 4 and FIG. 5. As shown in FIG. 5, the method 500 includes steps 510, 520, and 530. Step 510 and step 520 may be implemented by the analysis unit 420, and step 530 may be implemented by the processing unit 430. The method 500 may start at step 510.

[0075] In step 510, first angular data of M frames and second angular data of N frames prior to the M frames are obtained. Both M and N are positive integers.

[0076] In some embodiments, N and M may be different or may be the same.

[0077] In some embodiments, the M frames may be frames to be processed, so M may be predefined by the system 300 or inputted by a user.

[0078] In some embodiments, N is greater than or equal to a fourth threshold. The fourth threshold is a predefined value that indicates the minimum number of frames prior to the M frames and the minimum number of frames is required for processing the M frames. For example, the fourth threshold may be 32, and N may be 32, 40, 48, or the like. In these embodiments, the M frames may be processed by an image stabilization strategy corresponding to one or more motion parameters in a case where N is greater than or equal to the fourth threshold. The details of the one or more motion parameters that may be obtained according to the M frames and N frames will be introduced in step 520, and the details of the image stabilization strategy corresponding to the one or more motion parameters will be introduced in step 520 and step 530. The details of determining the corresponding image stabilization strategy will be introduced in later embodiments with reference to FIG. 7.

[0079] According to the foregoing embodiments where N is greater than or equal to the fourth threshold, the usage of the fourth threshold provides a condition to determine whether there are sufficient frames prior to the M frames. There are sufficient frames prior to the M frames (that is, N is greater than or equal to the fourth threshold), so that the M frames may be processed by the image stabilization strategy corresponding to the one or more motion parameters that are obtained according to the M frames and N frames. Therefore, the M frames may be processed by a more suitable image stabilization strategy.

[0080] In some other embodiments, N is smaller than the fourth threshold, which indicates that there are insufficient frames prior to the M frames. For example, the fourth threshold may be 32, and N may be 31 or 20. In these embodiments, the one or more motion parameters may not be obtained according to the M frames and N frames, or the one or more motion parameters obtained according to the M frames and N frames may not be accurate. Therefore, the M frames may be processed by a default image stabilization strategy. The details of the default image stabilization strategy will also be introduced in later embodiments with reference to FIG. 7.

[0081] According to the foregoing embodiments where N is smaller than the fourth threshold, there are not sufficient frames prior to the M frames (that is N is smaller than the fourth threshold), and it suggests that the uncertainty is high in this situation, so that the M frames may not be processed by the image stabilization strategy corresponding to the one or more motion parameters that are obtained according to the M frames and N frames, but rather by the default image stabilization strategy. Therefore, the M frames may be processed by a more suitable image stabilization strategy.

[0082] In some embodiments, the M frames may be continuous or non-continuous, or the N frames may be continuous or non-continuous. When the M frames or N frames are non-continuous, the M frames or N frames may be pulled from the continuous frames.

[0083] In some embodiments, a preprocess may be performed prior to step 510. The preprocess may be implemented by the preprocessing unit 410 in FIG. 4. The preprocessing unit 410 may receive raw motion data from the input unit 310 and frames from the input unit 310, and align timestamps of the raw motion data with timestamps of frames, so that each angle set in the raw motion data corresponds to a frame at the corresponding timestamp. After preprocessing, the angle set that has the same timestamp as the frame may refer to one angular data.

[0084] It is noted that in this disclosure, each angle set in the first angular data corresponds to one frame of the M frames, and each angle set in the second angular data corresponds to one frame of the N frames.

[0085] After obtaining the first angular data and the second angular data, the method commences at step 520.

[0086] In step 520, based on the first angular data and the second angular data, one or more motion parameters of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured are determined.

[0087] As described above, step 520 may be performed by the analysis unit 420. In detail, step 520 may be implemented by the feature extraction unit 421 in the analysis unit 420. In this case, the feature extraction unit 421 obtains the first angular data and the second angular data from the preprocessing unit 410 and determines the one or more motion parameters. The one or more motion parameters may describe different aspects of motions of the camera during the period from the time at which the first frame of the N frames is captured to the time at which the last frame of the M frames. For example, the one or more motion parameters may include an angular velocity, which describes the rate of rotation of the camera around its axes. The detailed process of determining the one or more motion parameters will be introduced in subsequent / later embodiments with reference to FIG. 6.

[0088] After determining the one or more motion parameters, the method commences at the step 530.

[0089] In step 530, the M frames are processed using an image stabilization strategy corresponding to the one or more motion parameters of the M frames.

[0090] As described above, step 530 may be performed by the processing unit 430. In detail, the M frames may be performed by the image stabilization unit 432 and the image warp unit 433 in the processing image unit 430. The image stabilization unit 432 may correct the motion trajectory of the M frames, and the image warp unit 433 may map the positions of pixels of the M frames to new locations according to the corrected frame trajectory. Therefore, the M frames may be transformed to a more stabilized version of frames.

[0091] According to the foregoing embodiments, the first angular data of the M frames and the second angular data of the N frames are obtained, where the N frames are prior to the M frames; the one or more motion parameters of the camera during the period from the time at which the first frame of the N frames is captured to the time at which the last frame of the M frames is captured are determined based on the first angular data and the second angular data; and the M frames is processed using the image stabilization strategy corresponding to the one or more motion parameters. The one or more motion parameters during the period from the time at which the first frame of the N frames is captured to the time at which the last frame of the M frames is captured are a more precise camera motion characterization that aligns with the current capturing scenario, the M frames may be processed accurately by the image stabilization strategy corresponding to the one or more motion parameters, and the stabilization performance on the M frames may be improved. Therefore, using this method may have more optimal stabilization performance. Furthermore, since the image stabilization strategy for processing the frames may change with the one or more motion parameters of the frames, this method may provide the image stabilization performance well in a variety of scenarios.

[0092] The image processing method including steps 510, 520, and 530 is described in the foregoing embodiments with reference to FIG. 4 and FIG. 5. Subsequently, a detailed description of step 520 is described in combination with FIG. 6.

[0093] As shown in FIG. 6, a process 600 for extracting features is depicted, and the process 600 corresponds to step 520 in FIG. 5.

[0094] The process 600 may be implemented by the feature extraction unit 421.

[0095] As shown in FIG. 6, the process 600 may start after step 510, where the first angular data and the second angular data are obtained. The features, which refer to the one or more motion parameters as shown in the dashed box, are determined in the process 600.

[0096] In some embodiments, the one or more motion parameters include at least one of: the first angular data and the second angular data, an angular velocity, an angular acceleration, standard deviation of angle, a kurtosis of velocity, a max distance of upper peaks, a skewness of angle, a velocity peak, a kurtosis of acceleration, a standard deviation of velocity or an acceleration peak of the camera during the period from the time at which the first frame of the N frames is captured to the time at which the last frame of the M frames is captured, a recent velocity value or a sine of angle of the camera when capturing the last frame in the M frames.

[0097] The above motion parameters may represent different perspectives of camera motions, and the descriptions of the above motion parameters are given as follows.

[0098] The first angular data and the second angular data have been explained in the embodiments corresponding to FIG. 5 and are not reiterated herein.

[0099] It is noted that each angular data may include at least one of rotation angle along a pitch axe, a rotation angle along a yaw axe, or a rotation angle along a roll axe. Therefore, the following formulas may be applied to calculate values for the rotation angles along the pitch axis, the yaw axis, and the roll axis respectively. That is, the value for the pitch axis may be determined, the value for the yaw axis may be determined, and the value for the roll axis may be determined.

[0100] The angular velocity refers to the rate of change of the first angular data and the second angular data. The formula for calculating the angular velocity is given as:

[0101] ang_i - ang_{i-1}

[0102] ang_vel_i = (ang_i - ang_{i-1}) / dt

[0103]

[0104] where ang_velj denotes the value of angular velocity of angle, i denotes an order number i∈[1,..., i,..., M+N], angj and ang^ denote an i-th and i-lth angles in the angular data, dt is a notation of a time difference between two consecutive time points, and the formula for calculating dt is given as:

[0105] dt = t(j) - t(j - 1),

[0106] where t(j) denotes the time at index j, and t(j — 1) denotes the time at the previous index at index j - 1.

[0107] The angular acceleration refers to the rate of change of the first angular data and the second angular data. The formula for calculating the angular acceleration is given as:

[0108] ang_velj - ang_veli-1

[0109] ang_acci = - — - where ang_accj denotes the value of angular acceleration of angle, i denotes an order number i∈[1,..., i,..., M+N], ang_velj and ang_veli-1denote an i-th and i-lth values of angular velocity in the angular data, dt is a notation of a time difference between two consecutive time points.

[0110] The standard deviation of angle refers to the amount of variation or dispersion in the first angular data and the second angular data. The formula for calculating the standard deviation of angle is given as:

[0111]

[0112] standard deviation of angle = √(Σ(ang_i - μ_ang)² / K)

[0113] where ang_std denotes the standard deviation of angle, i denotes an order number, i∈[1,..., i,..., M+N], angj denotes an i-th angle in the angular data, pangdenotes the mean value of the angular data, and K represents the value of M+N.

[0114] The formula for calculating μ_ang is given as: _ S angj

[0115] Hang -N

[0116] The kurtosis of velocity refers to a statistical measure indicating a shape and peakedness of a velocity distribution. The formula for calculating the kurtosis of velocity is given as:

[0117] ,.c,. K^Cang-veh - bvei)4

[0118] kurtosis of velocity = - 2

[0119] [ K^Cang-veli - pvei)2]

[0120] where vel_kurt denotes the kurtosis of velocity, ang_vel_i denotes the angle velocity of the i-th angle of the angular data, and μ_vel denotes the mean value of the angular velocity.

[0121] The formula for calculating μ_vel is given as:

[0122] > 2 ang-velj

[0123] Pvei -

[0124]

[0125] N

[0126] The formula for calculating ang_vel_i is given as:

[0127] angj - angi-1

[0128] ang_velj =

[0129] dt

[0130] where dt is a notation of a time difference between two consecutive time points.

[0131] The max distance of upper peaks refers to the maximum distance between peaks in the angular data. The formula for calculating the max distance of upper peaks is given as:

[0132] / ang_up_peaks\

[0133] max distance of upper peaks = max I - — - J

[0134]

[0135] where the ang_up_peaks denotes one or more angular data of the M and N frames where derivatives of these angular data change the sign from plus to minus.

[0136] The skewness of angle refers to a measure of the asymmetry of the angular data distribution. The formula for calculating the skewness of angle is given as:

[0137] V^angi - Pang)'

[0138] skewness of angle = - [VKS(angi - Hang ]

[0139] The velocity peak refers to the highest value in the velocity. The formula for calculating the velocity peak is given as:

[0140] velocity peak = max (ang_vel)

[0141] The kurtosis of acceleration refers to a measure describing the shape and tails of the acceleration distribution. The formula for calculating the kurtosis of acceleration is given as:

[0142] .. „ zSOng-acCi - Pace)4

[0143] kurtosis of acceleration = - 2

[0144] [1 / K S^anS—acci - Hacc)2]

[0145] where μ_acc denotes the mean value of the angular acceleration, and the formula for calculating μ_acc is given as:

[0146] _ E ang_acci

[0147] Hacc where ang_acc_i denotes the angular acceleration of the i-th angle of the angular data, and the formula for calculating the ang_acc_i is given as:

[0148] ang_vel_i - ang_vel_{i-1}

[0149] ang_accj = - — -

[0150]

[0151] The standard deviation of velocity refers to a measure of the amount of variation or dispersion in the velocity data. The formula for calculating the standard deviation of velocity is given as:

[0152] £(ang_veli - pvel)2

[0153] standard deviation of velocity = - - -K

[0154] The acceleration peak of the camera refers to the highest acceleration value. The formula for calculating the acceleration peak of the camera is given as:

[0155] acceleration peak = max (ang_acc)

[0156] The above parameters describe one or more aspects of motions of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured.

[0157] In addition, the sine of angle refers to the sine function of a rotation angle when a camera captures the last frame in the M frames. The formula for calculating the sine of angle is given as:

[0158] sine of angle = sin(angk)

[0159] where sin() denotes a function that calculates the sine of an angle, and k = K.

[0160] The recent velocity value refers to the most recent velocity of the camera when capturing the last frame in the M frames. The formula for calculating the recent velocity value is given as:

[0161] recent velocity = ang_velk

[0162] where k = N.

[0163] According to the foregoing embodiments, the one or more motion parameters includes various kinds of parameters for describing motions of the camera capturing frames from different perspectives. Therefore, the one or more motion parameters may provide a more precise camera motion characterization, which may help the determination of the image stabilization to process frames.

[0164] The process 600 of determining one or more motion parameters is described in the foregoing embodiments with reference to FIG. 6. After determining the one or more motion parameters, a subsequent process is to determine, based the one or more motion parameters, whether a target motion type of the M frames is determined and process the M frames using the image stabilization strategy corresponding to the target motion type in a case where the target motion type is determined. This subsequent process may be performed by the motion classification unit 422 in the analysis unit 420 and the selection unit 431 in the processing unit 430. The motion type may refer to a type of camera motion, where camera motions of the same type may share similar motions. For example, when the camera holder is jogging or running, the camera motions in these two scenarios may include small angle, high-frequency jitters, therefore these two motions may be of the same motion type. It is understood, the motion parameters of the camera motions of the same type may also share similarities. Therefore, frames corresponding to a motion type may be processed using an image stabilization strategy corresponding to the motion type.

[0165] According to the foregoing embodiments, it is determined whether the target motion type of the M frames is determined, and in a case where the target motion type is determined, the image stabilization strategy corresponding to the target motion type may be used to process the M frames. Because patterns of blurs may be similar for frames within the same motion type, a corresponding image stabilization strategy applied on frames within the same motion type may provide a simpler and more generalized way to process the M frames.

[0166] In some embodiments, the motion type of the M frames may be determined using a classification model. Therefore, determining whether the target motion type of the M frames is determined includes: obtaining an output of a classification model by inputting the one or more motion parameters into the classification model, where the classification model is trained based on a plurality of sample motion parameters and a plurality of motion types of the plurality of sample motion parameters; determining, based on the output, whether the target motion type of the M frames is determined.

[0167] The classification model may be a machine learning model, a statistical model, or other models capable of classification data, which is not limited herein. The plurality of sample parameters may refer to motion parameters described in the foregoing embodiments corresponding to FIG. 6. The motion types may be determined in advance for the plurality of sample parameters.

[0168] According to the foregoing embodiments, the one or more motion parameters are inputted into the classification model, and it is determined whether the target motion type of the M frames is determined, based on the output of the classification model. For example, the classification model outputs the target motion type directly or the classification model outputs probability information and the target motion type may be determined based on the probability information. Using the output of the classification model to determine whether the target motion type of the M frames is determined, this method may have better robustness.

[0169] In some embodiments, the output includes a plurality of probabilities of the plurality of motion types; and determining, based on the output, whether the target motion type of the M frames is determined includes: determining a first motion type of the plurality of motion types as the target motion type in a case where a probability of the first motion type is greater than a first threshold. According to the embodiments, the output includes the plurality of probabilities of the plurality of motion types; and the first motion type of the plurality of motion types is determined as the target motion type in a case where the probability of the first motion type is greater than the first threshold. The probabilities may indicate the certainty degrees of motion type classification results, where the probability is higher, and the certainty degree is higher, so that the first motion type having the probability greater than the first threshold may be determined as the target motion type, and the image stabilization strategy corresponding to the target motion type may be more suitable to process frames.

[0170] In some embodiments, determining, based on the output, whether the target motion type of the M frames is determined further includes: determining that the target motion type is not determined, in a case where the plurality of probabilities are all less than or equal to the first threshold.

[0171] According to the embodiments, in a case where there is no probability that is greater than the first threshold, which may indicate a high uncertainty of the classification result, and all the motion types may not be target suitable for frames, so that target motion type may not be determined. Compared with determining the target motion type regardless of whether there is a probability greater than the first threshold, not determining the target type may help avoid inaccurate determination of the target motion type and processing the frames using the subsequent inaccurate image stabilization strategy.

[0172] The following introduces the details of determining whether the target motion type of the M frames is determined, and processing the M frames using the image stabilization strategy where the image stabilization strategy is determined according to whether the target motion type of the M frames is determined.

[0173] The process 700 may start at step 710, which includes obtaining an output of the classification model by inputting the one or more motion parameters into the classification model. The process 700 may commence to step 720, where the target motion type of the M frames may be determined based on the output of the classification model.

[0174] In some embodiments, the output of the classification model may include a plurality of probabilities of the plurality of motion types. It is noted that the term target motion type refers to the motion type of the M frames. Each probability of a plurality of probabilities may be any numerical value, but the summation of the values of all probabilities should be a constant value. For example, in the case of three motion types, the plurality of probabilities may be as 0.8, 0.1, and 0.1, respectively, and the summation of the values of all probabilities is 1.

[0175] Then, a first motion type is determined as the target motion type if the probability of the first motion type is greater than a first threshold. For example, the plurality of probabilities may be 0.8, 0.1, and 0.1 respectively, the first threshold may be 0.7. In this case, the probability value 0.8 is greater than the first threshold of 0.7, therefore the first motion type that corresponds to the probability value 0.8 is determined as the target motion type.

[0176] In some embodiments, the target motion type of the M frames may not be determined if the plurality of probabilities of the plurality of motion types are all less than or equal to the first threshold. For example, the plurality of probabilities may be as 0.3, 0.3, and 0.4 respectively, the first threshold may be 0.7, and the summation of the values of all probabilities is 1. In this case, the probability values 0.3, 0.3, and 0.4 are all less than or equal to the first threshold of 0.7, therefore the target motion type is not determined.

[0177] The foregoing embodiments described the process 700, which may include determining the motion type of the M frames. In some embodiments, there may be a plurality of motion types and a plurality of image stabilization strategies corresponding to the plurality of motion types.

[0178] In some embodiments, the plurality of motion types may include a hand still motion, a huge directed motion, or a high-frequency motion. It is noted that the method provided in this disclosure may also apply to other motion types, which are not specifically limited in the present disclosure.

[0179] The hand still motion may represent the camera having small-angle, low-frequency jitters. For example, a hand still motion may occur when the person holding the camera is sitting in a chair or standing still, and the hand holding that camera stays relatively stable.

[0180] The huge directed motion may represent the camera having a large angle and low-frequency jitters. For example, the huge directed motion may occur when the person holding the camera is standing still, and capturing the scene of a playing child in the playground to record some memories. In this scenario, the hand holding that camera is moving over a huge angle but does not exhibit high-frequency jitters.

[0181] The high-frequency motion may represent the camera having small angle, high-frequency jitters. For example, the high-frequency motion may occur when the person holding the camera is running or walking, so the hand holding the camera exhibits short-angle, high-frequency jitters due to body movement.

[0182] According to the embodiments, three motion types including hand still motion, huge directed motion, and high-frequency motion are described. These three motion types may occur frequently in different camera shooting scenarios, therefore an image processing method that uses different image stabilization strategies corresponding to these three motion types may be applied across a wide range of scenarios.

[0183] FIG. 8 provides some exemplary visualizations of hand still motion, huge directed motion, and high-frequency motion. There are three curves in FIG. 8, which are denoted as curve 1, curve 2, and curve 3. The x-axis of the three curves is the order number of frames, and the y-axis of the curve 1, curve 2 and curve 3 are the values of angular data, angular velocity and angular acceleration, respectively.

[0184] The curves represent the trend of angular domain, angular velocity domain, and angular acceleration domain, respectively. In addition, each point on the curves may represent a value of angular, angular velocity, and angular at a time point.

[0185] Referring to the curve 1 in FIG. 8, the curve segment corresponding to hand still motion exhibits a low frequency of change and small amplitude. The curve segment corresponding to huge directed motion exhibits a low frequency of change but with a large amplitude. The curve segment corresponding to high-frequency motion exhibits a high frequency of change but with a small amplitude.

[0186] Referring to the curve 2 and curve 3, it may be observed that high-frequency motions cause higher numerical excitations in the domains of angular velocity and angular acceleration, while huge directed motion is more recognizable in the angular domain and angular velocity domain. Therefore, it may be concluded from FIG. 8 that different Therefore, it may be concluded from FIG. 8 that different motion parameters (angular, angular velocity, angular acceleration, etc.) may represent different aspects of camera motion and are effective in characterizing some types of motion. Consequently, utilizing one or more motion parameters may help more effectively classify motion types, thereby aiding in the determination of a more accurate image stabilization strategy.

[0187] In some embodiments, the hand still motion, huge directed motion, and high-frequency motion may be defined as follows. In the hand still motion, an angular value of the M frames is less than a second threshold, and a jitter frequency of the M frames is less than a third threshold; In the huge directed motion, the angular value of the M frames is greater than or equal to the second threshold, and the jitter frequency of the M frames is less than the third threshold; and In the high-frequency motion, the angular value of the M frames is greater than or equal to the second threshold, and the jitter frequency of the M frames is less than the third threshold.

[0188] In some other embodiments, a motion type may be determined, based on the one or more parameters, without a classification model. For example, a heuristic approach may be utilized, where one or more predefined thresholds may be used to compare with the values of motion parameters, to determine a motion type. The motion type corresponding to the M frames may be determined in other ways, which is not specifically limited in the embodiments of the present disclosure.

[0189] In some embodiments, the image stabilization strategy corresponding the one or more motion parameters may be determined based on values of the one or more motion parameters without determining a motion type. For example, the image stabilization strategy may be determined based on the relationship(s) between numerical values of the one or more motion parameters and a preset threshold.

[0190] The following describes the detail of the image processing method with reference to FIG.

[0191] 7. As shown in FIG. 7, there are three motion types: hand still motion, a huge directed motion, or a high-frequency motion, and the output of the classification model, i.e., the output of the motion classification unit 422, includes three probabilities, denoted as P(S) representing the probability that the motion type is hand still motion, P(D) representing the probability that the motion type is huge directed motion, and P(F) representing the probability that the motion type is high-frequency motion.

[0192] In addition, there are four image stabilization strategies: denoted as image stabilization (DF), Image stabilization (S), image stabilization (D), and image stabilization (F). Image stabilization (DF) corresponds to the situation that which the motion type is not determined. Image stabilization (S), image stabilization (D), and image stabilization (F) correspond to hand still motion, a huge directed motion, or a high-frequency motion, respectively.

[0193] The image stabilization strategies for processing frames may include using a filtering technique, a deep learning technique, and a homography matrices to reduce the blur of the frames, and other techniques that have image stabilization functionality, which is not specifically limited in the embodiments of the present disclosure.

[0194] In some embodiments, image stabilization (DF), image stabilization (S), image stabilization (D), and image stabilization (F) may be the same technique with different parameter settings. In these embodiments, the parameters of each image stabilization strategy are configured for stabilizing the frames that correspond to the motion type of the image stabilization strategy.

[0195] In these embodiments, image stabilization (DF) may have the parameters that are the average values of the corresponding parameters in one or more image stabilization strategies other than the default image stabilization strategy.

[0196] In some embodiments, image stabilization (DF), image stabilization (S), image stabilization (D), and image stabilization (F) may be four different techniques. In some other embodiments, some of the four image stabilization strategies may be the same technique with different parameters, and some of the four image stabilization strategies may be different techniques. For example, image stabilization (DF) may be using a filtering technique, and image stabilization (S), image stabilization (D) and image stabilization (F) may be using machine learning techniques with different parameters.

[0197] In these embodiments, image stabilization (DF) may be a strategy that has equal performance in processing frames captured by the camera across all motion types. It is understood that the parameters of the default image stabilization strategy may be defined in other ways, which is not specifically limited in the embodiments of the present disclosure.

[0198] Referring to FIG, 7, if the motion type of the M frames determined in 720 is hand still motion, image stabilization (S) is used to process the M frames; if the motion type of M frames is huge directed motion, image stabilization (D) is used to process the M frames; if the motion type of M frames is high-frequency motion, image stabilization (F) is used to process the M frames. In addition, if the motion type of M frames is not determined, image stabilization (DF) is used to process the M frames.

[0199] The image processing method according to the embodiments of the present disclosure is described above in combination with FIGS. 1 to 8. The image processing apparatus of the embodiments of the present disclosure will be described below in combination with FIGS. 9 and 10.

[0200] FIG. 9 is a block diagram of a structure of an image processing apparatus in accordance with some embodiments of the present disclosure. The image processing apparatus may be configured to perform the foregoing method embodiments, and therefore can also achieve beneficial effects of the foregoing method embodiments. In the embodiments of the present disclosure, the image processing apparatus may be an electronic device or a module of the electronic device (for example, a chip or a circuit).

[0201] As shown in FIG. 9, the image processing apparatus 900 includes an obtaining unit 910 and a processing unit 920. The image processing apparatus 900 is used to implement the foregoing method embodiments as shown in FIG. 5.

[0202] The obtaining unit 910 may be configured to obtain first angular data of M frames and second angular data of N frames prior to the M frames, where the first angular data includes one or more rotation angles of a camera when capturing the M frames, the second angular data includes one or more rotation angles of the camera when capturing the N frames, and M and N are positive integers. The processing unit 920 may be configured to determine, based on the first angular data and the second angular data, one or more motion parameters of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured; and process, based on the one or more motion parameters, the M frames using an image stabilization strategy corresponding to the one or more motion parameters.

[0203] In some embodiments, the one or more motion parameters include at least one of: the first angular data and the second angular data, an angular velocity, an angular acceleration, a standard deviation of angle, a kurtosis of velocity, a max distance of upper peaks, a skewness of angle, a velocity peak, a kurtosis of acceleration, a standard deviation of velocity or an acceleration peak of the camera during the period, a recent velocity value or a sine of angle of the camera when capturing the last frame in the M frames.

[0204] In some embodiments, the processing unit 920 is configured to determine, based the one or more motion parameters, whether a target motion type of the M frames is determined; and process the M frames using the image stabilization strategy corresponding to the target motion type in a case where the target motion type is determined.

[0205] In some embodiments, the processing unit 920 is configured to process the M frames using a default image stabilization strategy in a case where the target motion type is not determined.

[0206] In some embodiments, the obtaining unit 910 is configured to obtain an output of a classification model by inputting the one or more motion parameters into the classification model, where the classification model is trained based on a plurality of sample motion parameters and a plurality of motion types of the plurality of sample motion parameters; and the processing unit 920 is configured to determine, based on the output, whether the target motion type of the M frames is determined.

[0207] In some embodiments, the output includes a plurality of probabilities of the plurality of motion types; and the processing unit 920 is configured to determine a first motion type of the plurality of motion types as the target motion type in a case a probability of the first motion type is greater than a first threshold.

[0208] In some embodiments, the processing unit 920 is configured to determine that the target motion type is not determined, in a case the plurality of probabilities are all less than or equal to the first threshold.

[0209] In some embodiments, the target motion type is a hand still motion, a huge directed motion, or a high-frequency motion; and in the hand still motion, an angular value of the M frames is less than a second threshold, and a jitter frequency of the M frames is less than a third threshold; in the huge directed motion, the angular value of the M frames is greater than or equal to the second threshold, and the jitter frequency of the M frames is less than the third threshold; and in the high-frequency motion, the angular value of the M frames is less than the second threshold, and the jitter frequency of the M frames is greater than or equal to the third threshold.

[0210] In some embodiments, N is greater than or equal to a fourth threshold.

[0211] A more detailed description of functions of the obtaining unit 910 and processing unit 920 can be referred to the relevant description in the foregoing method embodiment shown in FIG. 5.

[0212] As shown in FIG. 10, an electronic device 1000 includes a processor 1010 and an interface circuit 1020. The processor 1010 and the interface circuit 1020 are coupled to each other. It is understood that the interface circuit 1020 can be a transceiver or an input / output interface. In some embodiments, the electronic device 1000 may further include a memory 1030, and the memory 1030 is configured to store instructions that, when executed by the processor 1010, cause the electronic device 1000 to perform any of the methods described above. The memory 1030 is further configured to store input data required by the processor 1010 to when it runs the instructions or the data produced after the processor 1010 has run the instructions.

[0213] In some other examples, the electronic device 1000 includes a processor 1010, and the interface circuit 1020 may also be understood as part of the processor 810.

[0214] When the electronic device 1000 is used to implement the method shown in FIG. 5, it may be interpreted as an electronic device itself, a module in the electronic device, a circuit or chip, or a combination thereof. The processor 1010 is used to implement the functions of the processing unit 920, and the interface circuit 1020 is used to implement the functions of the obtaining unit 910. For example, the electronic device 1000 may further include a memory 1030, the processor 1010 and the interface circuit 1020 are connected to the memory 1030 over a circuit or a wire, and the processor 1010 is configured to read and execute instructions stored in the memory 1030.

[0215] In the present disclosure, the terms “a” or “an” are defined to mean “at least one”, that is, these terms do not exclude a plural number of items, unless stated otherwise.

[0216] In the present disclosure, unless stated otherwise, the terms “connected” and “coupled”, and derivatives and variants thereof, refer herein to any structural or functional connection or coupling, either direct or indirect, between two or more elements. For example, connection or coupling between the elements can be acoustical, mechanical, optical, electrical, thermal, logical, or any combinations thereof.

[0217] In the present disclosure, the expression “based on” is intended to mean “based at least partly on”, that is, this expression can mean “based solely on” or “based partially on”, and so should not be interpreted in a limited manner. More particularly, the expression “based on” could also be understood as meaning “depending on”, “representative of’, “indicative of’, “associated with” or similar expressions.

[0218] In the present disclosure, the terms “system” and “network” may be used interchangeably in different embodiments of this disclosure. “At least one” means one or more, and “a plurality of’ means two or more. The term “and / or” describes an association relationship of associated objects, and indicates that three relationships may exist. For example, A and / or B may indicate the following three cases: Only A exists, both A and B exist, and only B exists, where A and B may be singular or plural. The character “ / ” indicates an “or” relationship between associated objects. “At least one of the following items (pieces)” or a similar expression thereof indicates any combination of these items, including a single item (piece) or any combination of a plurality of items (pieces). For example, “at least one of A, B, or C” includes: only A; only B; only C; A and B; A and C; B and C; or A, B, and C, and “at least one of A, B, and C” may also be understood as including: only A; only B; only C; A and B; A and C; B and C; or A, B, and C. In addition, unless otherwise specified, ordinal numbers such as “first” and “second” in embodiments of this disclosure are used to distinguish between a plurality of objects, and are not used to limit a sequence, a time sequence, priorities, or importance of the plurality of objects.

[0219] A person skilled in the art should understand that embodiments of this disclosure may be provided as a method, an apparatus (or system), a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), or a computer program product. Therefore, this disclosure may use a form of a hardware-only embodiment, a software-only embodiment, or an embodiment with a combination of software and hardware. Moreover, this disclosure may use a form of a computer program product that is implemented on one or more computer-usable storage media (including but not limited to a disk memory, an optical memory, and the like) that includes computer-usable program code.

[0220] This disclosure is described with reference to the flowcharts and / or block diagrams of the method, the device (system), and the computer program product according to this disclosure. It should be understood that computer program instructions may be used to implement each process and / or each block in the flowcharts and / or the block diagrams and a combination of a process and / or a block in the flowcharts and / or the block diagrams. The computer program instructions may be provided for a general-purpose computer, a dedicated computer, an embedded processor, or a processor of another programmable data processing device and enable a machine to execute the instructions. When executed by any computer or the processor of a programmable data processing device, the instructions cause the apparatus to implement specific functions as described in one or more procedures in the flowcharts and / or one or more blocks in the block diagrams. The computer program instructions may alternatively be stored in a computer-readable memory that can indicate a computer or another programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate an artifact that includes an instruction apparatus. The instruction apparatus implements a specific function in one or more procedures in the flowcharts and / or one or more blocks in the block diagrams.

[0221] The computer program instructions may alternatively be loaded onto a computer or another programmable data processing device, so that a series of operations and steps are performed on the computer or the another programmable device, so that computer-implemented processing is generated. Therefore, the instructions executed on the computer or on another programmable device provide steps for implementing specific functions as described in one or more procedures in the flowcharts and / or one or more blocks in the block diagrams.

[0222] It is clear that a person skilled in the art can make various modifications and variations to this disclosure without departing from the scope of this disclosure. This disclosure is intended to cover these modifications and variations of this disclosure provided that they fall within the scope of protection defined by the following claims and their equivalent technologies.

Claims

CLAIMS1. An image processing method, comprising:obtaining first angular data of M frames and second angular data of N frames prior to the M frames, wherein the first angular data comprises one or more rotation angles of a camera when capturing the M frames, the second angular data comprises one or more rotation angles of the camera when capturing the N frames, and M and N are positive integers;determining, based on the first angular data and the second angular data, one or more motion parameters of the camera during a period from a time at which a first frame of the N frames is captured to a time at which a last frame of the M frames is captured; andprocessing, based on the one or more motion parameters, the M frames using an image stabilization strategy corresponding to the one or more motion parameters.

2. The method of claim 1, wherein the one or more motion parameters comprise at least one of: the first angular data and the second angular data, an angular velocity, an angular acceleration, a standard deviation of angle, a kurtosis of velocity, a max distance of upper peaks, a skewness of angle, a velocity peak, a kurtosis of acceleration, a standard deviation of velocity or an acceleration peak of the camera during the period, a recent velocity value or a sine of angle of the camera when capturing the last frame in the M frames.

3. The method of claim 1 or 2, wherein processing, based on the one or more motion parameters, the M frames using the image stabilization strategy corresponding to the one or more motion parameters, comprises:determining, based the one or more motion parameters, whether a target motion type of the M frames is determined; andprocessing the M frames using the image stabilization strategy corresponding to the target motion type in a case where the target motion type is determined.

4. The method of claim 3, further comprising:processing the M frames using a default image stabilization strategy in a case where the target motion type is not determined.

5. The method of claim 3 or 4, wherein determining whether the target motion type of the M frames is determined comprises:obtaining an output of a classification model by inputting the one or more motion parameters into the classification model, wherein the classification model is trained based on a plurality ofsample motion parameters and a plurality of motion types of the plurality of sample motion parameters;determining, based on the output, whether the target motion type of the M frames is determined.

6. The method of claim 5, wherein the output comprises a plurality of probabilities of the plurality of motion types; and determining, based on the output, whether the target motion type of the M frames is determined comprises:determining a first motion type of the plurality of motion types as the target motion type in a case where a probability of the first motion type is greater than a first threshold.

7. The method of claim 6, wherein determining, based on the output, whether the target motion type of the M frames is determined further comprises:determining that the target motion type is not determined, in a case where the plurality of probabilities are all less than or equal to the first threshold.

8. The method of any one of claims 3 to 7, wherein the target motion type is a hand still motion, a huge directed motion, or a high-frequency motion; andin the hand still motion, an angular value of the M frames is less than a second threshold, and a jitter frequency of the M frames is less than a third threshold;in the huge directed motion, the angular value of the M frames is greater than or equal to the second threshold, and the jitter frequency of the M frames is less than the third threshold; and in the high-frequency motion, the angular value of the M frames is less than the second threshold, and the jitter frequency of the M frames is greater than or equal to the third threshold.

9. The method of any one of claims 1 to 8, wherein N is greater than or equal to a fourth threshold.

10. An image processing apparatus comprising units for performing the method according to any one of claims 1 to 9.

11. An electronic device comprising processing circuitry for performing the method according to any one of claims 1 to 9.

12. A chip, comprising an input / output (I / O) interface and a processor, wherein the processoris configured to call and run a computer program stored in a memory, to enable a device installed with the chip to perform the method according to any one of claims 1 to 9.

13. An electronic device, comprising:one or more processors; anda memory storing instructions which, when executed by the one or more processors, cause the device to perform the method of any one of claims 1 to 9.

14. A computer-readable storage medium having instructions stored thereon which, when executed by a device, cause the device to perform the method of any one of claims 1 to 9.

15. A computer program product storing instructions which, when executed, cause an apparatus to perform the method of any one of claims 1 to 9.

Images