Image blur correction device, method for controlling same, program, and storage medium
Patent Information
- Application Number
- PCT/JP2026/009766
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2026-01-15
- Filing Date
- 2026-03-12
- Publication Date
- 2026-09-24
Smart Images
Figure JP2026009766_24092026_PF_FP_ABST
Abstract
Description
Image blur correction apparatus, control method therefor, program, and storage medium
[0001] The present disclosure relates to a technique for correcting image blur in an imaging apparatus.
[0002] In recent years, various blur correction functions have been proposed for correcting image blur caused by shake such as camera shake applied to imaging apparatuses such as digital cameras. By mounting these functions in an imaging apparatus, more favorable captured images can be acquired.
[0003] As one method for correcting camera shake, a technique called electronic image stabilization is known, which generates an image free of camera shake by performing geometric transformation processing on a captured frame image to cancel the movement of camera shake. Furthermore, methods for measuring camera shake for performing electronic image stabilization include a method that uses motion information of the imaging apparatus obtained from a gyro sensor mounted on the imaging apparatus, and a method that uses motion information between temporally consecutive frame images by analyzing the captured frame images.
[0004] Here, the motion information of the imaging apparatus obtained from the gyro sensor is angular velocity information in the yaw, pitch, and roll directions of the imaging apparatus. A geometric transformation matrix is generated from these pieces of information and sequentially integrated, thereby obtaining a geometric deformation amount for image blur correction between frames. At this time, in order to perform image blur correction with higher accuracy, it is necessary to integrate the geometric transformation matrices with as fine a time resolution as possible. However, when the number of integrations increases, a large amount of error included in each individual geometric transformation matrix accumulates, which may conversely adversely affect the video after blur correction.
[0005] In the imaging apparatus described in Patent Document 1, when accumulating motion information of the imaging apparatus obtained from a gyro sensor, accumulation of errors is suppressed by resetting an accumulated value if the accumulated time exceeds a predetermined threshold.
[0006] Japanese Patent Application Laid-Open No. 2017-26667
[0007] However, in Patent Document 1, a reset process is performed when the cumulative time exceeds a threshold, resulting in discontinuities in the cumulative value of the gyro data before and after the reset. Therefore, discontinuities may also occur in the image after image blur correction has been performed using this data.
[0008] This disclosure has been made in view of the above-mentioned problems and provides an image blur correction device that can make discontinuities less likely to occur in images that have been image blur corrected. The image blur correction device according to this disclosure comprises: an acquisition means for acquiring information on the movement of an imaging device; a generation means for generating geometric transformation matrices representing the amount of image blur at regular time intervals based on the acquired information on the movement of the imaging device; an integration means for integrating a plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate an integrated geometric transformation matrix; and a geometric deformation means for correcting image blur by performing geometric deformation processing on an image captured by the imaging device based on the integrated geometric transformation matrix, wherein the generation means sets the regular time interval based on information on the amount of error of the geometric transformation matrices stored in advance in a storage means.
[0009] Furthermore, the image blur correction device relating to this disclosure comprises: acquisition means for acquiring information on the movement of an imaging device; generation means for generating geometric transformation matrices representing the amount of image blur at regular time intervals based on the acquired information on the movement of the imaging device; integration means for integrating a plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate an integrated geometric transformation matrix; and geometric deformation means for performing geometric deformation processing on an image captured by the imaging device based on the integrated geometric transformation matrix to correct the image blur, wherein the generation means sets the regular time interval based on the magnitude of the movement of the imaging device.
[0010] According to this disclosure, discontinuities can be made less likely to occur in images with image blur correction.
[0011] Other features and advantages of this disclosure will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are given the same reference numeral.
[0012] The attached drawings are included in the specification and constitute part thereof, illustrating embodiments of the present disclosure and used together with the description to explain the principles of the present disclosure. Block diagram showing the configuration of an imaging device relating to the first embodiment of the present disclosure. Flowchart showing the operation of the imaging device in the first embodiment. Diagram illustrating the effect of rotation order Diagram illustrating the effect of rotation order in correction Timing chart of blur correction processing Timing chart of blur correction processing Diagram showing the relationship between error amount and time resolution Diagram showing the relationship between error amount and time resolution Flowchart showing the operation of an imaging device in the second embodiment. Diagram showing a method for setting time resolution using a blur reference value.
[0013] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessary, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.
[0014] (First Embodiment) Figure 1 is a block diagram showing the configuration of an imaging device 100 according to the first embodiment of the present disclosure.
[0015] In Figure 1, the optical system 101 collects light from the subject and forms an image of the subject. The image sensor 102 is an element that converts the image of the subject formed by the optical system 101 into photoelectric data, and is composed of a CCD sensor or a CMOS sensor. Here, the optical system 101 is equipped with a blur correction mechanism to correct image blur caused by the shaking of the imaging device 100.
[0016] The image processing unit 103 forms a video signal from the electrical signal output from the image sensor 102. The image processing unit 103 includes an A / D conversion circuit (not shown), an auto gain control circuit (AGC), and an auto white balance circuit (AWB) to form a digital signal. The image sensor 102 and the image processing unit 103 constitute an imaging system for acquiring images. One or more frames of the video signal formed by the image processing unit 103 are stored in the memory 104 or used for subsequent processing or display.
[0017] The motion information acquisition unit 105 acquires motion information caused by camera movement and camera shake occurring in the imaging device 100, for example, using motion detection sensors such as a gyro sensor or an acceleration sensor. The time resolution calculation unit 106 calculates the time resolution when the geometric transformation matrix is integrated in the subsequent blur amount calculation unit 107.
[0018] The blur amount calculation unit 107 converts the motion information of the imaging device 100 obtained from the gyro sensor into the form of a geometric transformation matrix, performs integration processing on these, and stores it in the memory 104. The blur correction amount calculation unit 108 calculates the geometric deformation amount for image blur correction using the geometric transformation matrix obtained from the blur amount calculation unit 107 and the integrated integrated geometric transformation matrix stored in the memory 104.
[0019] The geometric deformation unit 109 performs geometric deformation processing to correct image blur using the geometric deformation amount calculated by the blur correction amount calculation unit 108. The image with the blur corrected is then displayed on a display device by the video output unit 110 or stored in an image storage device (not shown).
[0020] The CPU 112, which consists of a microcomputer, controls the entire imaging device 100. In Figure 1, the signal lines connecting the CPU 112 and the various components of the imaging device 100 are not shown, but the CPU 112 is connected to the various components of the imaging device 100 by an internal bus (not shown) that enables it to exchange signals with each component. The memory 104 stores the control program executed by the CPU 112, as well as control constants and variables.
[0021] The operation of the imaging device 100 configured as described above will be explained using the flowchart shown in Figure 2.
[0022] In step S201, the CPU 112 converts the subject image formed by the optical system 101 into an analog signal corresponding to the subject's brightness using the image sensor 102, and generates a video signal through processing by the development processing unit 103. The development processing unit 103 converts the analog signal into, for example, a 14-bit digital signal using an A / D conversion unit (not shown). Furthermore, the digital video signal, which has undergone signal level correction and white level correction by an AGC (Auto Gain Control) circuit and an AWB (Auto White Balance) circuit (not shown), is stored in the memory 104. In the imaging device 100 of this embodiment, frame images are generated sequentially at a predetermined frame rate, and the transmitted and stored frame images are also updated sequentially.
[0023] In step S202, the CPU 112 uses the motion information acquisition unit 105 to acquire motion information occurring in the imaging device 100. Here, the motion information of the imaging device 100 in this embodiment refers to the temporal changes in the position and orientation of the imaging device, and includes not only movements unintended by the photographer, such as camera shake, but also movements intentional to the photographer, such as panning and tilting. In this embodiment, a gyro sensor is assumed as the means for acquiring the motion information of the imaging device, and the movement of the imaging device 100 itself is acquired as angular velocity information in the yaw, pitch, and roll directions. The motion information acquired in step S202 is transmitted to the time resolution calculation unit 106 and the blur amount calculation unit 107.
[0024] In step S203, the CPU 112 reads the lookup table (hereinafter referred to as LUT) relating to the error amount of the geometric transformation quantity stored in memory 104 and transmits it to the time resolution calculation unit 106. Details of the LUT stored in memory 104 will be explained in step S204 below.
[0025] In step S204, the CPU 112 calculates the time resolution of the calculation process performed by the subsequent blur amount calculation unit 107, using the LUT for the error amount obtained from the memory 104, or using the motion information of the imaging device obtained by the motion information acquisition unit 105 and the LUT for the error amount obtained from the memory 104.
[0026] Here, we will first explain the errors that occur when calculating the amount of blur, which is the issue addressed in this disclosure. The motion information acquisition unit 105 acquires motion information occurring in the imaging device 100 from the gyro sensor mounted on the imaging device 100. The motion information here refers to the rotational motion in the yaw, pitch, and roll directions occurring in the imaging device 100, and can be obtained as an angular quantity obtained by integrating the angular velocities related to each rotation axis. Furthermore, in the subsequent geometric transformation unit 109, in order to perform geometric transformation processing for blur correction on the frame image, it is necessary to convert this motion information into the form of a geometric transformation matrix. For this purpose, the angular quantities in the yaw, pitch, and roll directions are expressed in the form of rotation matrices, and these are combined into a single geometric transformation matrix that can represent the geometric transformation amounts of the three rotation axes. The rotation matrices representing the geometric transformations in the yaw, pitch, and roll directions are shown below.
[0027]
[0028] Here, Hy, Hp, and Hr in equations 1 to 3 are rotation matrices for the yaw, pitch, and roll directions, respectively, and α, β, and γ represent the amounts of rotation angles for the yaw, pitch, and roll directions, respectively. By combining the rotation matrices from equations 1 to 3, a geometric transformation matrix representing the geometric deformation amount for each rotation direction is generated. However, a phenomenon occurs where different geometric transformation matrices are generated depending on the order of combination. The order of combination refers to the order in which the geometric transformations of rotation are performed during the geometric transformation process. For example, when the imaging device 100 rotates in the order of yaw, pitch, and roll, the geometric transformation matrix Hypr is calculated as shown in the equation below.
[0029]
[0030] As shown in the above equation, the rotation matrix for the yaw direction, which is rotated first, is on the right, and multiplying so that the direction of rotation that is rotated later is on the left determines the order of rotation. As another example, the geometric transformation matrix Hryp when rotated in the order of roll, yaw, and pitch is calculated as follows, similar to equation 4.
[0031]
[0032] From equations 4 and 5, different rotation orders result in different geometric transformation matrices as a result of the synthesis. Therefore, even if geometric transformation processing is actually performed on frame images using these geometric transformation matrices, the same image will not be produced; instead, images with different transformations will be generated. This is because, while yaw, pitch, and roll movements occur simultaneously in the actual movement of the imaging device, the geometric transformation processing is performed in a convenient order for processing purposes. Furthermore, although there are six possible combinations of the synthesis order of yaw, pitch, and roll, none of them strictly represent the actual movement of the imaging device. The difference between the image captured by the actual movement of the imaging device at that time and the image generated by geometric transformation processing using the calculated geometric transformation matrices is the amount of error that is the subject of this case.
[0033] The matters explained so far using formulas will now be explained again with diagrams, and the error quantities targeted in this embodiment will be further explained. Figure 3 shows the case when rotation occurs around the x and y axes from a certain state, and Figure 4 shows the case when the rotation is canceled from state C in Figure 3.
[0034] In Figure 3, first consider the case where the coordinate system is in the state shown on the far left (State: A). From here, consider the case where the phenomenon occurs in the order of a 90-degree rotation around the y-axis (State: B) → a 90-degree rotation around the x-axis (State: C) (upper transition in Figure 3), and the case where the phenomenon occurs in the order of a 90-degree rotation around the x-axis (State: B') → a 90-degree rotation around the y-axis (State: C') (lower transition in Figure 3). Comparing states C and C' in Figure 3, it is clear that the orientation of the coordinate system differs depending on the order in which the phenomenon occurs. In the case of an imaging device, the orientation of the imaging device differs.
[0035] It is important to note here that both the upper and lower panels of Figure 3 contain the same information: a 90-degree rotation around the x-axis and a 90-degree rotation around the y-axis. Therefore, it is not possible to determine whether the state is C or C' simply by observing the angles. This is the physical significance of equations 4 and 5, which show that different rotation orders result in different geometric transformation matrices as a result of the composition.
[0036] Next, we consider the process of returning to the original state (equivalent to vibration isolation in the case of an imaging device). Figure 4 shows an example of the process applied. Figure 4 shows the same state as Figure 3, with states A, B, and C. At state C, we know that the device has rotated 90 degrees around the x and y axes from the original state (state A). Therefore, we show the cases where the device is returned in the order of -90 degree rotation around the x axis (state D) → -90 degree rotation around the y axis (state E), and the cases where it is returned in the order of -90 degree rotation around the y axis (state D') → -90 degree rotation around the x axis (state E'). The return from state C → state D → state E, which is done by tracing the order of the effects in reverse, is correct. That is, state A and state E are equivalent, and vibration isolation is working correctly. On the other hand, the return from state C → state D' → state E', which is done in the wrong order, is not correct. As is clear from the figure, state A and state E' are not equivalent, so vibration isolation is not working.
[0037] Figures 3 and 4 illustrate an extreme example where 90-degree rotations are applied sequentially, clearly demonstrating the impact of incorrect rotation order. This is the amount of error we are addressing in this case. As will be described later, this error is reduced by accumulating multiple geometric transformation matrices generated at regular time intervals over a predetermined time to produce an integrated geometric transformation matrix.
[0038] One method to suppress errors that occur in the geometric transformation matrix representing such fluctuations is to shorten the time interval during which the geometric transformation matrix is calculated, in other words, to increase the time resolution.
[0039] Here, we define time resolution. In this embodiment, time resolution is defined as the number of times the geometric transformation matrix is calculated per unit time (the reciprocal of the time interval for calculating the geometric transformation matrix). A fine, high, or large time resolution means that the time interval for calculating the geometric transformation matrix is short. Conversely, a coarse, low, or small time resolution means that the time interval for calculating the geometric transformation matrix is long.
[0040] The finer the time resolution, the smaller the values of the angular quantities α, β, and γ used in the above calculations become. This means that regardless of the rotation order used, the result will approach the identity matrix equally, thus reducing the difference in the geometric transformation matrix due to differences in rotation order. In other words, it becomes possible to suppress the amount of error that is the subject of this case, as explained earlier.
[0041] Here, Figure 5A shows a timing chart schematically representing the sequence of steps in a conventional image stabilization process. In this figure, the time interval 301 represents the shooting period for one frame, and the image stabilization process is basically performed using this period as a single unit. Gyroscope data representing the movement of the imaging device 100 is continuously output from the gyro sensor mounted on the imaging device 100, and gyrooscope data at some point during the one-frame period is extracted to calculate a geometric transformation matrix, and these are then integrated.
[0042] The geometric transformation matrix calculated at this time represents the motion of the imaging device 100 over a certain period of time, and in this figure, it corresponds to the motion over one frame period. Then, by integrating the geometric transformation matrices each representing motion over one frame period (this integration is to obtain the amount of movement from the start time of imaging, not the integration of geometric transformation matrices within one frame described later), the motion of each frame for the captured video can be expressed as a concatenation. In the present embodiment, this process sequentially accumulates changes in the posture of the imaging device, and this information is expressed as an integrated geometric transformation matrix. In this figure, gyro data is extracted at the timing when image reading is completed as shown at 302, but the present invention is not limited thereto; other timings such as the exposure center timing of imaging may also be used, and there is no problem as long as the timing is the same for each frame. Then, after calculation of the geometric transformation matrix, integration processing of the geometric transformation matrix, blur correction amount calculation, and geometric transformation processing are sequentially executed. As described above, in conventional blur correction processing, calculation of the geometric transformation matrix and geometric transformation are only performed once per frame, which increases the possibility that errors in the geometric transformation matrix caused by the aforementioned synthesis of rotation matrices affect the result of the geometric transformation processing.
[0043] In contrast, FIG. 5B shows a timing chart of blur correction processing in the present embodiment. As shown at 303 in this figure, in the present embodiment, gyro data is acquired a plurality of times with fine time resolution within one frame period (=one geometric transformation), and calculation of the geometric transformation matrix is also performed with fine time resolution in accordance therewith. Then, as shown at 304 in this figure, a result obtained by integrating geometric transformation matrices at a fixed timing during the frame period is extracted (the geometric transformation matrices are integrated during the one frame period, which corresponds to generation of an integrated geometric transformation matrix by an integrating means), and the result is transmitted to the subsequent blur correction amount calculation processing. By adopting this configuration, the rotation amounts of yaw, pitch, and roll used for calculating a single geometric transformation matrix can be reduced, so it becomes possible to suppress errors in the geometric transformation matrix caused by rotation order.
[0044] As described so far, in the calculation of a geometric transformation matrix, the greater the rotation amounts of yaw, pitch and roll used, the larger the error that occurs in the geometric transformation matrix. Therefore, the time resolution for calculating the geometric transformation matrix is set based on information relating to the error amount of the geometric transformation matrix stored in advance in a storage means, or is changed according to the magnitude of shake occurring in the imaging apparatus 100, whereby the error amount can be controlled. In the case of changing according to the magnitude of shake, particularly by making the time resolution finer as shake increases, the error amount that is the target in the present case can be suppressed.
[0045] Here, the magnitude of motion of the imaging apparatus 100 in the present embodiment refers to the amplitude and frequency magnitude of motion obtained from a gyro sensor; the greater the amplitude, the greater the motion of the imaging apparatus, and furthermore, the greater the frequency, the greater the motion of the imaging apparatus is determined to be.
[0046] However, the finer the time resolution for integrating geometric transformation matrices is made, the more this leads to an increase in cumulative error due to integration processing. The geometric transformation matrix calculated from gyro data includes various errors other than the error due to the rotation order described above, and an example thereof is noise inherent in the gyro data itself caused by influences such as temperature change, electrical interference, and aging deterioration. These errors are mainly removed by performing noise reduction processing after acquiring gyro data in the motion information acquisition unit 105, but there are cases where they cannot be completely removed and remain residual.
[0047] Another example is the quantization error that occurs when processing gyro data and calculating geometric transformation matrices. In order to perform various calculations within the electronic circuit of the imaging device 100, the angular velocity information of the analog signal obtained from the gyro sensor is converted into a digital signal, and rounding is performed to a fixed number of decimal places for fixed-point conversion. The difference between the value obtained from the original gyro sensor and the result of these processes becomes the error in the geometric transformation matrix. Since these errors are present individually in each calculated geometric transformation matrix, when the geometric transformation matrices are accumulated, these errors are also accumulated. Therefore, the finer the time resolution, the more geometric transformation matrices are calculated and accumulated during one frame period, and the more times they are accumulated, the greater the error in the accumulated geometric transformation matrix becomes. Consequently, in order to minimize the overall error of the geometric transformation matrix used for image blur correction, it is necessary to set a time resolution that minimizes the sum of both the error in the geometric transformation matrix due to the rotation order (combination order) and the error due to accumulation as much as possible.
[0048] In this embodiment, the time resolution of the geometric transformation matrix integration process is set using LUTs related to these errors so as to minimize the amount of error that occurs when calculating the blur correction amount. Furthermore, these LUTs related to the errors of the geometric transformation matrix are stored in memory 104 and are read from memory 104 in step S203 described above and transmitted to the time resolution calculation unit 106.
[0049] Here, if we denote the total error amount calculated from the error in the geometric transformation matrix caused by the rotation order and the cumulative error due to the integration process as E, then the relationship between them can be expressed as shown in Equation 6 below.
[0050] E = A / α1 + α1B (Equation 6) In the above equation, the error amount E is the total error amount for each frame period, A is the error amount of the geometric transformation matrix due to the rotation order per frame period, and B is the error amount of each geometric transformation matrix that is a factor in the cumulative error. α1 is the time resolution of the geometric transformation matrix within one frame period, and indicates the number of times the integration process is performed within one frame period.
[0051] From Equation 6, the error A of the geometric transformation matrix due to the rotation order decreases as the amount of angle used in the calculation of the geometric transformation matrix decreases. For example, if the integration process is performed only once in one frame period, α1 becomes 1, and the error amount is A. If the integration process is performed twice in one frame period, α1 becomes 2, and the amount of angle used in the calculation of the geometric transformation matrix is the amount of angle over half the frame period, so it becomes A / 2. Thus, it can be said that the error of the geometric transformation matrix due to the rotation order is inversely proportional to the number of integrations, that is, to the time resolution.
[0052] Furthermore, the cumulative error of the geometric transformation matrix is proportional to the number of times the geometric transformation matrix is multiplied. For example, if the time resolution α1 is 1, the cumulative error is the error amount B for one geometric transformation matrix. If the time resolution α1 is 2, the multiplication process is performed twice during one frame, so the errors of each geometric transformation matrix are multiplied by two, meaning the cumulative error can be expressed as 2B.
[0053] From the above, in order to suppress the overall error amount E as much as possible, it is necessary to set the time resolution α1 such that E in Equation 6 is minimized. Assuming general blurring, the time resolution α1 that minimizes E in Equation 6 can be determined (which is generally finer than the time resolution of the geometric deformation), and this information can be stored in memory 104 and used.
[0054] Furthermore, although the imaging device 100 has various shooting settings, the time resolution α1 may be determined by referring to the settings related to blur. In other words, in modes that prioritize blur suppression (for example, settings that strongly apply electronic blur correction), there is a high possibility of large blurs occurring, so it is convenient to set a finer time resolution for integrating the geometric transformation matrix.
[0055] Furthermore, the error amount A of the geometric transformation matrix due to the rotation order varies depending on the magnitude of the movement of the imaging device 100. When the movement of the imaging device is large, the angular quantity used to calculate the geometric transformation matrix is also large, so the error amount A is large. Conversely, when the movement of the imaging device is small, the error amount A is small. Thus, since the value of the error amount A in equation 6 changes according to the magnitude of the movement of the imaging device, in order to find the time resolution α1 that minimizes the overall error amount E, it is necessary to observe the movement of the imaging device and set an appropriate value for the error amount A.
[0056] The magnitude of the imaging device's movement can be determined using the angle information of the imaging device obtained from the gyro sensor, but the corresponding error amount must be measured in advance. One method for measuring the error amount is to calculate the difference between the known movement obtained by applying a known movement to the imaging device using a vibration table, etc., and the movement obtained from the geometric transformation matrix calculated within the imaging device. In addition, by changing the amplitude and frequency of the known movement applied to the imaging device and comprehensively measuring the error amount for movements tailored to anticipated use cases, the information necessary for the LUT can be obtained.
[0057] The format of the LUT could, for example, store the error amount for the magnitude of movement in the yaw, pitch, and roll directions, or it could use another method, such as calculating the norms of the rotation angles in the yaw, pitch, and roll directions and associating the error amount with their magnitudes. Using the norms of the rotation angles in three directions can simplify subsequent calculations and save memory capacity for storing the LUT.
[0058] Similarly, it is necessary to store the amount of error B contained in each geometric transformation matrix that contributes to the cumulative error. The amount of error caused by the gyro sensor can be measured by referring to the specifications of the gyro sensor or by comparing the output values of the gyro sensor when the imaging device is stationary and when a known movement is applied. Furthermore, it is possible to determine the amount of error of the geometric transformation matrix from the number of significant digits when calculating the geometric transformation matrix, and by adding this to the amount of error of the gyro sensor mentioned above, it becomes possible to obtain the amount of error B contained in each geometric transformation matrix and store it in memory 104.
[0059] Figure 6A shows an example of a graph of Equation 6, created based on the magnitude of the error amount obtained from the LUT stored in memory 104. In this figure, the vertical axis represents the magnitude of the overall error amount E in Equation 6, and the horizontal axis represents the time resolution when integrating the geometric transformation matrix. Also, 401 in this figure shows the curve drawn when Equation 6 is represented on a graph.
[0060] Curve 401 shows a large value when the time resolution α1 is small, and the value decreases as the time resolution α1 increases, but increases again when the time resolution increases even further. This is due to the following reasons. First, when the time resolution α1 is small, the error amount A of the geometric transformation matrix due to the rotation order becomes large, and as the time resolution α1 increases, the amount of angle used in the calculation of the geometric transformation matrix becomes smaller, and the error amount A also decreases. In addition, as the time resolution α1 increases, the cumulative error of the error amount B due to the integration process increases, but up to a certain point, the decrease in the error amount A is greater, so the error amount E decreases. Then, at the time resolution shown in 402 of this figure, the error amount E is minimized, but thereafter, the increase in the cumulative error of the error amount B exceeds the decrease in the error amount A, so the error amount E increases. In other words, by setting the time resolution α1 to the value shown in 402, it is possible to suppress the amount of error that occurs in the geometric transformation matrix to the greatest extent possible.
[0061] Furthermore, Figure 6B shows an example of a graph of Equation 6 when an error amount of a different magnitude than that in Figure 6A is obtained from the LUT stored in memory 104. As shown in this figure, when the values of error amounts A and B are different, the overall error amount E traces a different curve 403 than curve 401. The time resolution α1 at which the error amount E is minimized is the value shown in 404 of the same figure, which is larger than the value shown in 402 of Figure 6A. This indicates that because the error amount A in Figure 6B is larger than the error amount A in Figure 6A, more time resolution is required before it falls below the cumulative error increase of error amount B.
[0062] As described above, by setting the time decomposition of the geometric transformation matrix integration process based on the magnitude of the imaging device's movement, it becomes possible to suppress errors that occur in the geometric transformation matrix.
[0063] Another method for setting the time resolution α1 is to compare the magnitudes of yaw, pitch, and roll movements. If the magnitudes of movement in one of the three rotational directions are significantly smaller than those in the other two directions, the time resolution can be reduced (the time interval can be increased). This is because, if only the movement in one rotational direction is large, the rotation matrices representing the movements in the other two rotational directions can be considered as identity matrices, and the geometric transformation matrices obtained from equations 4 and 5 above can be treated as the same. In other words, in such cases, errors in the geometric transformation matrices due to differences in the rotation order do not occur, and there is no need to increase the time resolution (make the time interval finer). The thresholds for determining the magnitude of each rotational direction in this case can be determined in advance based on methods such as the motion information of the imaging device acquired from the gyro sensor when the imaging device is stationary or subjected to minute vibrations.
[0064] Another setting method involves, for example, setting the settings based on the calculations performed in subsequent processes such as the shake amount calculation process and the shake correction amount calculation process. When calculating the shake amount, the time required for calculation varies depending on, for example, how complex the shake amounts to be corrected. If only translational shake is included, the amount of calculation required is small, but if tilt shake and non-linear shake are also included in the correction target, a larger amount of calculation will be required.
[0065] Similarly, when calculating the amount of image stabilization, the amount of computation required to calculate the stabilization amount varies depending on the complexity of the camera work being handled. For example, if the camera is held handheld and stationary, there is no camera movement, so the stabilization amount can be calculated with less computation. However, when shooting while walking, panning, tilting, or tracking a specific subject, a much larger amount of computation will be required.
[0066] In motion image blur correction, if the series of processes required to correct the blur cannot be completed within one frame period, problems such as delays will occur between the image displayed in live view and the image actually being captured. This makes comfortable shooting difficult. Therefore, it is possible to prevent processing delays by estimating the calculations required for the series of processes for blur correction in advance and setting a time resolution that ensures that the processing for one frame is completed reliably within one frame period. Here, several methods for setting the time resolution when integrating geometric transformation matrices have been described, but it is expected that a more appropriate time resolution can be set not only by using each method individually, but also by combining multiple methods.
[0067] In step S205, the CPU 112 uses the blur amount calculation unit 107 to calculate the blur amount in the form of a geometric transformation matrix using the motion information of the imaging device 100 obtained from the motion information calculation unit 105 and the time resolution value of the integration process obtained from the time resolution calculation unit 106.
[0068] Here, the motion information of the imaging device refers to the rotation angle amounts in the yaw, pitch, and roll directions of the imaging device obtained from the gyro sensor mounted on the imaging device. Since these rotation angle amounts are expressed in Euler angles, it is first necessary to convert these Euler angles into the form of a rotation matrix. The Euler angle values representing the rotation angles in the yaw, pitch, and roll directions can be converted as shown in equations 1, 2, and 3 above. Next, by combining these rotation matrices, a single geometric transformation matrix representing the motion in the three rotation directions is generated. As shown in equations 4 and 5, the geometric transformation matrix generated will differ depending on the order in which the rotation matrices are combined. However, since the calculation and integration of the geometric transformation matrix is performed based on the time resolution obtained from the time resolution calculation unit 106, the influence of these differences can be reduced, so there is no problem regardless of the order in which they are combined.
[0069] In this case, setting the time resolution periodically, for example at the start of one frame period, allows for adaptive changes in the time resolution in accordance with changes in the movement of the imaging device, resulting in better image stabilization. On the other hand, if there is a special purpose such as reducing the amount of computation or power consumption, the time resolution set at the start of shooting may be used continuously, or the time resolution may be reset at longer time intervals.
[0070] The calculated geometric transformation matrix is then integrated based on the time resolution obtained from the time resolution calculation unit 106. The geometric transformation matrix calculated from the rotation matrix represents the movement of the imaging device over a period determined by the set time resolution, and in this state, it is difficult to know exactly how the imaging device moved. Therefore, by integrating the geometric transformation matrices calculated for each period defined by the time resolution, it becomes possible to connect the movement of the imaging device in each period as a continuous movement, and it becomes possible to know what kind of trajectory the imaging device traced during the shooting period. The movement of the imaging device obtained in this way includes not only movements intended by the photographer, such as pan and tilt, but also unintended movements such as camera shake, and the magnitude of this unintended movement is the amount of blur occurring in the imaging device. The amount of blur calculated in this way is transmitted to the blur correction amount calculation unit 108.
[0071] In step S206, the CPU 112 uses the image stabilization amount calculation unit 108 to calculate the image stabilization amount for applying image stabilization processing to the captured image using the image stabilization information obtained from the image stabilization amount calculation unit 107.
[0072] The geometric transformation matrix transmitted from the shake amount calculation unit 107 contains both intentional and unintentional shake movements of the photographer. Generally, intentional movements of the photographer, such as camera work, are low-frequency movements, while unintentional shake movements, such as camera shake, are relatively high-frequency movements. Therefore, by accumulating the transmitted geometric transformation matrix in the time direction and applying a high-pass filter (hereinafter HPF) to it, it is possible to remove low-frequency movements, leaving only high-frequency shake movements. However, the geometric transformation matrix at this time is a composite of yaw, pitch, and roll movements. Therefore, even if an HPF is applied to each element of the geometric transformation matrix, each element contains a mixture of rotation matrices for each direction, making it impossible to correctly apply the HPF to each rotation direction individually. Accordingly, in order to correctly apply the HPF to each of the three rotation directions, it is necessary to extract the rotation amounts of yaw, pitch, and roll from the composite geometric transformation matrix. Here, we will explain how to calculate the rotation angles in the yaw, pitch, and roll directions from the geometric transformation matrix shown in Equation 4. First, each component of Equation 4 is expressed as shown below.
[0073]
[0074] At this time, the rotation angle amounts α, β, and γ in each rotation direction can be calculated as follows.
[0075] β = sin⁻¹h (Equation 8) α = sin⁻¹(-g / cosβ) (Equation 9) γ = cos⁻¹(e / cosβ) (Equation 10) In this embodiment, a method for calculating the rotation angle in each rotation direction from the geometric transformation matrix synthesized in the rotation order shown in Equation 4 has been described, but it is possible to calculate other rotation orders in the same way. Then, by applying HPF to the rotation angle amounts α, β, and γ obtained in this way, the amount of blur occurring in each rotation direction can be determined. After that, by synthesizing the extracted blur amounts in each rotation direction again, a geometric transformation matrix representing the amount of blur occurring in the imaging device 100 can be calculated. Since it is necessary to apply a geometric transformation process to the frame image in order to cancel out the motion of the geometric transformation matrix representing the amount of blur calculated in this way, the final blur correction amount is the inverse matrix of the geometric transformation matrix calculated here. The blur correction amount calculated as described above is transmitted to the geometric transformation unit 109.
[0076] In step S207, the CPU 112 uses the geometric transformation unit 109 to correct the blur in the captured video using the blur correction amount calculated by the blur correction amount calculation unit 108. The blur correction amount obtained from the blur correction amount calculation unit 108 is a geometric transformation matrix that can transform the entire screen in a way that cancels out the blur motion occurring in the captured frame image. Therefore, by applying geometric transformation processing to the frame images that are acquired sequentially as the camera is shot, using the blur correction amounts that are also acquired sequentially, it becomes possible to generate a good video with corrected blur.
[0077] The blur-corrected frame image obtained in the manner described above is transmitted to the video output unit 110. The video output unit 110 displays the blur-corrected frame image obtained from the geometric deformation unit 109 on monitors (not shown) or records and stores it in an image storage device.
[0078] As described above, in this embodiment, the time resolution for calculating and integrating the geometric transformation matrix is changed based on information regarding the error amount of the geometric transformation matrix that is stored in the storage means beforehand.
[0079] More preferably, the magnitude of the movement of the imaging device is observed, and the time resolution for calculating and integrating the geometric transformation matrix is changed according to that magnitude. This makes it possible to appropriately suppress both the error in the geometric transformation matrix due to differences in rotation order and the cumulative error due to the integration of the geometric transformation matrix, thereby reducing the overall error in the blur correction amount.
[0080] (Second Embodiment) Figure 7 is a flowchart showing the blur correction operation in the second embodiment of the present disclosure. In this embodiment, the time resolution for integrating the geometric transformation matrix is determined using a predetermined blur reference value. In this embodiment, only the parts that perform different processing from the first embodiment will be described in Figures 5A, 5B and 7.
[0081] Steps S201 and S202 in Figure 7 are the same as steps S201 and S202 in the flowchart of Figure 2.
[0082] In step S401, the CPU 112 uses the time resolution calculation unit 106 to compare the magnitude of the shake of the imaging device 100 obtained from the motion information acquisition unit 105 with the shake reference value stored in the memory 104. In this embodiment, the time resolution is determined based on a predetermined shake magnitude. This makes it possible to reduce the error in the geometric transformation matrix due to the rotation order to a magnitude equivalent to the error that occurs when a predetermined shake occurs.
[0083] Furthermore, in this embodiment, the blur reference value is set to the movement of the imaging device when it is held handheld and stationary. In actual shooting use cases, the amount of blur when shooting in a handheld, stationary state is considered to be among the smallest. Therefore, by setting the time resolution to match this amount of blur, it is possible to reduce the error in the geometric transformation matrix due to the rotation order to the size of the blur closest to the minimum. As for how to set the blur reference value, for example, one method is to measure the amount of blur that occurs in the imaging device in advance and store that amount as the blur reference value in memory 104. Other setting methods include, for example, using the amount of blur when the shooting state determination function separately installed in the imaging device determines that it is in a handheld, stationary state, or simply setting the amount of blur in a typical handheld, stationary state.
[0084] Here, if the motion of the imaging device obtained from the motion information acquisition unit 105 is even smaller than the blur reference value, the amount of error is smaller than the amount of error caused by the reference blur size, without needing to refine the time resolution. In such cases, even if the time resolution is refined, it would only lead to an unnecessary increase in the cumulative error of the geometric transformation matrix, so the process proceeds to the blur amount calculation process in step S205 without changing the time resolution. On the other hand, if the motion of the imaging device is larger than the blur reference value, it is determined that the time resolution needs to be refined, and the process proceeds to the resolution calculation process in step S402.
[0085] In step S402, the CPU 112 calculates the time resolution to be set using the motion information of the imaging device obtained from the motion information acquisition unit 105 and the blur reference value stored in the memory 104.
[0086] Here, we will explain the method for determining the temporal resolution using the graph shown in Figure 8, which schematically represents the magnitude of the blur.
[0087] In Figure 8, the vertical axis of each graph represents the magnitude of shake, and the horizontal axis represents time. Furthermore, 601 schematically represents an example of the temporal variation in the magnitude of camera shake when shooting handheld in a stationary state, and 602 schematically represents an example of the magnitude of camera shake when shooting while walking. In addition, 603 represents the time interval for estimating and integrating the geometric transformation matrix when shooting handheld in a stationary state, that is, the reciprocal of the time resolution, and is set, for example, at the timing of capturing frame images.
[0088] The magnitude of the shake in the handheld, stationary state (601) is used as the shake reference value to calculate the time resolution for hand shake in the walking state. Let's assume that the magnitude of hand shake in the walking state is five times greater than the magnitude of hand shake in the handheld, stationary state. In this case, as shown in 604 of Figure 8, if the time resolution for the movement of hand shake in the walking state is made five times finer than the time resolution for the handheld, stationary state, the magnitude of the shake that occurs during the period defined by each time resolution will be equivalent.
[0089] The amount of error in the geometric transformation matrix due to the order of synthesis is determined by the magnitude of the rotation angle used to calculate the geometric transformation matrix; therefore, the amount of error in the geometric transformation matrix calculated with the time resolution determined as described above will be the same. When actually calculating the time resolution, the time resolution to be set can be determined by calculating how many times the magnitude of the motion of the imaging device obtained from the motion information acquisition unit 105 corresponds to the blur reference value, and rounding that to an integer value.
[0090] By setting the time resolution in this way, it becomes possible to suppress the error amount of the geometric transformation matrix due to the synthesis order in the walking state to the same level as the error amount in the handheld stationary state. Furthermore, by using this setting method, a time resolution is set to be approximately the same as the reference amount of shake, and a time resolution higher than that is never set, thus suppressing the increase in cumulative error due to the integration of geometric transformation matrices. The time resolution calculated in this way is used in the processing of step S403.
[0091] In step S403, the CPU 112 uses the time resolution calculation unit 106 to determine an upper limit for the time resolution calculated in step S402. By calculating the time resolution using the method described above, it becomes possible to set a time resolution that is equivalent to the amount of blur used as a reference value, and as a result, it becomes possible to suppress the increase in the cumulative error of the geometric transformation matrix. However, if the movement of the imaging device is significantly large, such as when taking images while running, the time resolution calculated using the method described above may become too fine, and the cumulative error may become very large. Therefore, in step 403, the calculated time resolution is compared with a predetermined upper limit, and if the time resolution exceeds the upper limit, the upper limit is set as the time resolution to be actually used. As for how to set the upper limit of the time resolution, for example, it may be set based on the amount of error obtained from the difference between the geometric transformation matrix calculated in advance when the imaging device is stationary or when a known movement is performed using an excitation table, and the correct movement. The set upper limit is then stored in the memory 104. By doing so, it is possible to prevent the time resolution from becoming excessively fine, thereby minimizing the increase in cumulative error due to the integration of geometric transformation matrices.
[0092] Then, if the calculated time resolution is smaller than the upper limit, the process proceeds to step S404 and sets that time resolution as the time resolution for the blur amount calculation process. Otherwise, the process proceeds to step S405 and sets the upper limit value as the time resolution.
[0093] Steps S205 to S207 in Figure 7 are the same as steps S205 to S207 in the flowchart of Figure 2.
[0094] As explained above, in this embodiment, the time resolution for integrating the geometric transformation matrix is determined using a predetermined deviation reference value. This makes it possible to suppress the error amount of the geometric transformation matrix due to the order of synthesis to the same level as the error amount at the reference deviation value, while also minimizing the cumulative error of the geometric transformation matrix as much as possible.
[0095] (Other Embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (for example, an ASIC) that implements one or more functions.
[0096] This disclosure is not limited to the embodiments described above, and various modifications and alterations are possible without departing from the spirit and scope of the disclosure. Accordingly, the claims are attached to make the scope of the disclosure public.
[0097] This application claims priority based on Japanese Patent Application No. 2025-045619, filed on 19 March 2025, and Japanese Patent Application No. 2026-005234, filed on 15 January 2026, and all of the contents of those applications are incorporated herein by reference.
Claims
1. An image blur correction device comprising: an acquisition means for acquiring information on the movement of an imaging device; a generation means for generating geometric transformation matrices representing the amount of image blur at regular time intervals based on the acquired information on the movement of the imaging device; an integration means for integrating a plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate an integrated geometric transformation matrix; and a geometric deformation means for correcting image blur by performing geometric deformation processing on an image captured by the imaging device based on the integrated geometric transformation matrix, wherein the generation means sets the regular time interval based on information on the amount of error of the geometric transformation matrices stored in advance in a storage means.
2. The image blur correction device according to claim 1, characterized in that the constant time interval is set based on the settings related to the imaging device and information regarding the amount of error of the geometric transformation matrix stored in the storage means beforehand.
3. Image blur correction device comprising: acquisition means for acquiring information on the movement of an imaging device; generation means for generating geometric transformation matrices representing the amount of image blur at regular time intervals based on the acquired information on the movement of the imaging device; integration means for integrating a plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate an integrated geometric transformation matrix; and geometric deformation means for performing geometric deformation processing on an image captured by the imaging device based on the integrated geometric transformation matrix to correct image blur, wherein the generation means sets the regular time interval based on the magnitude of the movement of the imaging device.
4. The image blur correction device according to claim 3, characterized in that the generating means reduces the constant time interval as the movement of the imaging device increases.
5. The image blur correction device according to claim 3, characterized in that the generation means sets the constant time interval such that the error amount of the integrated geometric transformation matrix is minimized, based on information regarding the error amount of the geometric transformation matrix which is stored in the storage means in advance.
6. The image blur correction device according to claim 3, characterized in that the generating means sets the constant time interval such that the amount of error in the geometric transformation matrix for the actual amount of movement of the imaging device is equal in magnitude to the amount of error in the geometric transformation matrix for the amount of movement of the imaging device when it is being held by hand.
7. The image blur correction device according to claim 6, characterized in that the generating means sets the constant time interval such that the constant time interval does not become smaller than a predetermined value.
8. The image blur correction device according to claim 3, characterized in that the acquisition means acquires information on the pitch, yaw, and roll movements of the imaging device; the generation means generates three rotation matrices representing the geometric transformations due to the pitch, yaw, and roll movements of the imaging device, respectively; further generates a geometric transformation matrix representing the amount of blur in the image obtained by combining the three rotation matrices at regular time intervals; and the integration means integrates the plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate the integrated geometric transformation matrix.
9. The image blur correction device according to claim 8, characterized in that the generating means increases the constant time interval when at least two of the pitch, yaw, and roll movements of the imaging device are less than a predetermined amount, compared to when all of the pitch, yaw, and roll movements are greater than the predetermined amount.
10. The image blur correction device according to claim 9, characterized in that the generating means sets the constant time interval such that the sum of the error in the geometric transformation matrix resulting from the synthesis of the three rotation matrices and the error resulting from the integration of the geometric transformation matrices is minimized.
11. The image blur correction device according to claim 1 or 3, characterized in that the predetermined time is the duration of one frame.
12. The image blur correction device according to claim 3, characterized in that the generating means sets a certain time interval such that the calculation of the image blur correction amount is completed within the period of one frame, in accordance with the complexity of the method for calculating the image blur correction amount in the geometric deformation means.
13. A control method for an image blur correction device, comprising: an acquisition step of acquiring information on the movement of an imaging device; a generation step of generating geometric transformation matrices representing the amount of image blur at regular time intervals based on the acquired information on the movement of the imaging device; an integration step of accumulating a plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate an integrated geometric transformation matrix; and a geometric deformation step of performing geometric deformation processing on an image captured by the imaging device based on the integrated geometric transformation matrix to correct the image blur, wherein in the generation step, the regular time interval is set based on information on the amount of error of the geometric transformation matrices stored in advance in a storage means.
14. A control method for an image blur correction device, comprising: an acquisition step of acquiring information on the movement of an imaging device; a generation step of generating geometric transformation matrices representing the amount of image blur at regular time intervals based on the acquired information on the movement of the imaging device; an integration step of integrating a plurality of geometric transformation matrices generated at regular time intervals over a predetermined time to generate an integrated geometric transformation matrix; and a geometric deformation step of performing geometric deformation processing on an image captured by the imaging device based on the integrated geometric transformation matrix to correct image blur, wherein in the generation step, the regular time interval is set based on the magnitude of the movement of the imaging device.
15. A program for causing a computer to execute each step of the control method for an image blur correction device according to claim 13 or claim 14.
16. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method for an image blur correction device according to claim 13 or claim 14.