Imaging device, image processing method, and program
The imaging device stabilizes the clipping frame using past and future frames to maintain smooth tracking and framing, addressing abrupt changes and subject recognition loss in imaging devices.
Patent Information
- Application Number
- PCT/JP2026/015565
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-04-17
- Publication Date
- 2026-11-12
AI Technical Summary
Existing imaging devices struggle with abrupt and unnatural changes in the clipping frame when tracking a subject that suddenly changes movement or loses subject recognition, leading to delayed and erratic framing.
The imaging device stabilizes the clipping frame by generating it from subject recognition results and applying image stabilization using both past and future frames, ensuring smooth transitions and preventing out-of-frame references.
This approach results in a stable and smooth tracking video with natural framing changes, even when subject recognition is lost, by referencing future frames to predict and adjust framing accordingly.
Smart Images

Figure JP2026015565_12112026_PF_FP_ABST
Abstract
Description
Imaging device, image processing method, and program
[0001] The present technology relates to an imaging device, an image processing method, and a program, and particularly relates to clipping processing for a captured image.
[0002] Various types of imaging devices, such as digital cameras and smartphones equipped with an imaging function, include some that have a function of clipping and recording a part of a captured image. In particular, a process of tracking a target subject within a moving image frame and clipping the image of the target subject is known. Japanese Patent Application Laid-Open No. 2007-328746 listed below discloses a technology for tracking a target subject. Japanese Patent Application Laid-Open No. 2013-258468 listed below discloses a technology for camera shake correction for moving images.
[0003] JP 2007-328746 A JP 2013-258468 A
[0004] As a tracking framing function in an imaging device, there is a function in which a user pre-specifies a composition such as bust shot or full-body shot, so that a range such as a bust shot is clipped and recorded from each frame of a captured moving image. However, in this function, the clipping frame for framing is determined based on the observed subject recognition result and past information, so when the subject suddenly changes its movement, or when the subject recognition result cannot be obtained (when the subject is lost), it is not possible to follow with a correct clipping frame, and the frame change may become abrupt or unnatural.
[0005] In view of the above, an object of the present technology is to obtain a smooth clipped image that is not unnatural when tracking and clipping a specific subject as in a tracking framing function.
[0006] An imaging device according to the present technology includes an image processing unit that generates a clipping frame from a subject recognition result of a frame of captured image data, performs a stabilization process of stabilizing the clipping frame based on information of a plurality of frames ranging from past frames to future frames, and performs a clipping process on the frame of the captured image data using the clipping frame subjected to the stabilization process. The clipping frame for automatic tracking framing is stabilized using not only past information but also future information.
[0007] This is a block diagram of the imaging device in an embodiment of this technology. This is an explanatory diagram of the image stabilization and correction of automatic tracking framing in the embodiment. This is an explanatory diagram of the image stabilization and correction of the embodiment. This is an explanatory diagram of past and future information referencing in the embodiment. This is an explanatory diagram of the stabilization trajectory in the embodiment and comparative example. This is an explanatory diagram of the image stabilization in the embodiment and comparative example. This is an explanatory diagram of manual tracking cropping in the embodiment and comparative example. This is a flowchart of the automatic tracking framing in the embodiment.
[0008] The embodiments will be described below in the following order: <1. Imaging device configuration> <2. Stabilization of the cropping frame> <3. Processing example> <4. Summary and modified examples>
[0009] In this disclosure, "image" includes both video and still images. While the embodiments primarily assume video capture, the technology of this disclosure can also be applied to still image capture.
[0010] <1. Imaging Device Configuration> Figure 1 shows an example of the configuration of the imaging device 1 of the embodiment. The imaging device 1 can be implemented in the form of a digital camera, or as a smartphone or tablet device equipped with an imaging function.
[0011] The imaging device 1 comprises a control unit 30, an imaging unit 31, a display unit 32, an operation unit 33, a sensor unit 34, an image processing unit 35, a storage unit 36, and a memory unit 37. Note that the configuration in Figure 1 is merely an example showing the essential parts, and it may also include components not shown, and it is not necessary to include all of the components shown.
[0012] The imaging unit 31 includes a lens unit, an image sensor that receives incident light corresponding to one or more lenses of the lens unit, and a signal processing circuit for the photoelectric conversion performed by the image sensor. The lens unit in the imaging unit 31 includes an optical system such as a zoom lens, a focus lens, and an aperture mechanism, as well as a drive mechanism for these optical elements. Light from the subject is guided by this optical system and focused onto the image sensor.
[0013] The image sensor in the imaging unit 31 is configured as, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type. The image sensor performs processes such as CDS (Correlated Double Sampling) processing and AGC (Automatic Gain Control) processing on the electrical signal obtained by photoelectric conversion of the received light, and further performs A / D (Analog / Digital) conversion processing. The captured image data as digital data is then output to the image processing unit 35.
[0014] The image processing unit 35 is configured, for example, as an image processing processor. The image processing unit 35 performs processing on the captured image data that is captured and input by the imaging unit 31, such as Y / C development processing, codec processing, and file generation processing. The video and still image data output after processing by the image processing unit 35 is stored in a storage medium by the storage unit 36. As will be described later, the image processing unit 35 also performs cropping processing as automatic tracking framing. The image processing unit 35 is also provided with a memory area as a buffer memory 35a, which temporarily stores captured image data and various calculated values during the processing. The buffer memory 35a may be a memory built into the processor as the image processing unit 35, or it may be a separate memory chip.
[0015] The display unit 32 consists of a display panel and its display driving circuit, and displays various information to the user. The display panel is made up of a liquid crystal display (LCD) or an electroluminescent (EL) panel, etc.
[0016] The display unit 32 performs various displays on the display screen based on instructions from the control unit 30. For example, the display unit 32 can display a preview image (monitor image) taken during shooting, a playback image, or various icons and controls as part of the user interface.
[0017] The operation unit 33 comprehensively represents the input devices for the user to perform various operations. Specifically, these include physical controls such as shutter buttons, and touch panels that detect operations on controls (buttons, icons, sliders, etc.) displayed on the display unit 32. The operation unit 33 detects user operations, and signals corresponding to the input operations are sent to the control unit 30.
[0018] The sensor unit 34 refers to various sensors mounted on the imaging device 1. For example, the sensor unit 34 may include an IMU (inertial measurement unit), which can detect angular velocity using three-axis angular velocity (gyro) sensors (pitch, yaw, and roll) and acceleration using an acceleration sensor. The sensor unit 34 may also include an illuminance sensor. Furthermore, the sensor unit 34 may include a distance measuring sensor, such as a TOF (Time of Flight) sensor. Various information detected by the sensor unit 34, such as illuminance information, IMU data, and distance measuring data, is supplied to the control unit 30 and the image processing unit 35.
[0019] The storage unit 36 performs processing to record image files (content files) such as still image data and video data, as well as attribute information of image files, thumbnail images, etc., on a recording medium such as non-volatile memory. The actual form of the storage unit 36 can be diverse. For example, the storage unit 36 may be a circuit unit that performs recording and playback on flash memory built into the imaging device 1, or it may be a card recording and playback circuit that performs recording and playback access on a memory card (for example, portable flash memory) that can be attached to and removed from the imaging device 1. In addition, the storage unit 36 may be implemented as an HDD (Hard Disk Drive) or SSD (Solid State Drive) when built into the imaging device 1.
[0020] The control unit 30 is composed of a processor (arithmetic processing unit) such as a CPU (Central Processing Unit). The memory unit 37 stores information used by the control unit 30 for processing. This memory unit 37 may include, for example, ROM (Read Only Memory), RAM (Random Access Memory), or flash memory. The memory unit 37 may be a memory area built into the microcomputer chip that constitutes the control unit 30, or it may be composed of a separate memory chip.
[0021] The RAM in the memory unit 37 is used as a workspace for the CPU of the control unit 30 during various data processing operations, and is used for the temporary storage of data and programs. The ROM and flash memory (non-volatile memory) in the memory unit 37 are used to store the OS (Operating System) for the CPU of the control unit 30 to control each part, as well as application programs and various setting information for the processor of the control unit 30 or the image processing unit 35 to perform various operations.
[0022] The control unit 30 controls the entire imaging device 1 by executing programs stored in the ROM or flash memory of the memory unit 37. For example, the control unit 30 controls the zoom, focus, and aperture adjustment of the optical system of the imaging unit 31, controls the shutter speed of the image sensor, processes signals for captured image data, performs imaging and recording operations in response to user operations, and plays back recorded image files.
[0023] Although the image processing unit 35 and the control unit 30 are shown as separate components in the diagram, they may be formed in a single processor.
[0024] <2. Stabilization of the cropping frame> The cropping process as automatic tracking framing of this technology will be explained. Until now, in automatic tracking framing, the cropping frame was determined based on the subject recognition result observed in the captured image data and past information. As a result, if the subject suddenly changed its movement or if the subject recognition result could not be obtained (i.e., it was lost), it was not possible to track with the correct cropping frame.
[0025] Because the system uses subject recognition results from the current and past frames, attempting to make the tracking of the cropping frame smooth inevitably results in a delayed tracking. Also, when the subject is lost, the frame position is not determined, so it can only be estimated from past trajectories. However, when the system recovers from being lost, it hastily tries to track the subject, resulting in an unnatural frame trajectory.
[0026] Therefore, in this embodiment, the captured image data is buffered and recorded with a slight delay, allowing future subject recognition results to be referenced when determining the cropping frame. As a result, even if the subject moves irregularly, it becomes possible to generate a cropping frame (framing frame) that moves smoothly and without delay. In particular, since there is a possibility of losing the subject recognition results during the process of setting the cropping frame, being able to observe the future state after recovery from a temporary loss is effective in making the changes in the cropping frame follow a natural trajectory.
[0027] This will be explained in Figure 2. The upper part of Figure 2 shows one frame of past, present, and future image data labeled "Past," "Current," and "Future." Note that there are several frames between "Past" and "Current," and between "Current" and "Future."
[0028] In the subject recognition process for captured images, feature point detection is performed in each frame, targeting, for example, the skeletal points of a human body, to determine the pixel area in which a human body is captured. This allows for tracking of the human subject in each frame. However, in each frame, there is a shift in the frame frame due to the user's hand tremor during image capture. Even if the position and size of the subject within the frame do not change, the effects of hand tremor will still be present. Therefore, for example, the position of a certain feature point of the subject will change in each frame, as shown by arrow HS.
[0029] Based on the subject recognition results for each frame of the captured image data, an automatic tracking framing cropping frame 50 can be generated. By applying image stabilization to the cropping frame 50, the effects of blur can be reduced, as shown by the arrow HS in the middle of Figure 2. In other words, changes in the position of the subject's characteristic points PS due to blur can be reduced. In this case, by generating image stabilization information for the current frame using future frames in addition to past frames, more desirable image stabilization can be achieved than when using only past frames.
[0030] The cropping frame 50 is defined by its representative position and size. The size is, for example, the size in the x and y coordinate directions, and the representative position can be any information indicating a specific position within the cropping frame 50, such as the central coordinates of the subject area or specific skeletal coordinates. The following explanation will use an example where the representative position is the central coordinates. The central coordinates are the coordinate values where the center point of the cropping frame 50 is located within the x and y coordinates of the frame.
[0031] Here, the cropping frame 50 is further stabilized as shown in the lower part of Figure 2. This stabilization process maintains a stable frame trajectory even if the recognition of the human subject is lost, and also keeps the subject as close to the center of the cropping frame 50 as possible. Depending on the accuracy of subject recognition, the position of the cropping frame 50 may change slightly from frame to frame, and this process also eliminates such positional fluctuations. In other words, it is a process that stabilizes the composition of the image data captured by automatic tracking framing. This stabilization process, that is, the stabilization of the cropping frame 50 for the current frame, is performed by referring to information about the cropping frame 50 from the past to the future. This ensures that the size of the cropping frame and the central coordinates change smoothly over time.
[0032] The image data extracted using the stabilized cropping frame 50 is then recorded as video data. As a result, it is possible to obtain a stable video file with automatic tracking framing that prevents out-of-frame references, reduces the effects of image loss, and avoids abrupt changes in composition. Out-of-frame references occur when the area outside the outermost pixels of the image sensor is included within the frame, resulting in black areas where pixels do not exist within the frame.
[0033] Figure 3 shows how the output of image data used for recording is delayed when image stabilization and correction are performed by referring to future frames, along with the time-series change in position for each frame. The waveform in the upper panel shows the position change caused by camera shake in each frame. The rightmost position is assumed to be the most recent frame, and the position change due to camera shake continues from past frames. The middle panel shows the change in the position of the cropping frame 50 as a result of image stabilization in each frame, and the lower panel shows the change in the position of the cropping frame 50 due to stabilization.
[0034] Here, frame FRc is considered the current frame to be processed. Image stabilization information for frame FRc is generated by referencing x frames in the future and x frames in the past, i.e., ±x frames, from this frame FRc.
[0035] Let's explain using Figure 4. From the perspective of frame FRc, the actual latest frame, frame FRc+x, is a future frame. Therefore, if information for future frames x frames is buffered from the timing of frame FRc, then it is possible to reference the future x frames, i.e., from frame FRc+1 to frame FRc+x. Furthermore, for the past x frames of frame FRc, from frame FRc-1 to frame FRc-x, the image stabilization information Hc-x has already been calculated from the image stabilization information Hc-1. Therefore, for frame FRc, it is possible to generate image stabilization information Hc that references ±x frames at a timing with an x-frame delay.
[0036] By performing such calculations for each frame, image stabilization becomes possible at a timing delayed by x frames, as shown in the middle of Figure 3. Furthermore, the cropped frame 50 of the image-stabilized frame FRc is subjected to stabilization processing by referring to ±y frames.
[0037] In Figure 4, the cropped frame 50 corrected for camera shake using the image stabilization information Hc is defined as cropped frame CFc. With respect to cropped frame CFc, the cropped frames 50 corrected for camera shake in past y frames are cropped frame CFc-1 to cropped frame CFc-y. These have already been obtained. On the other hand, with respect to cropped frame CFc, the cropped frames 50 corrected for camera shake in future y frames are cropped frame CFc+1 to cropped frame CFc+y. Cropped frame CFc+y is obtained y frames after cropped frame CFc is obtained. At that y-frame later timing, stabilization processing can be performed on cropped frame CFc, referencing ±y frames.
[0038] Therefore, as shown in Figure 3, frame FRc is extracted with a delay time of x+y frames, which includes a delay of x frames for calculating image stabilization information and a delay of y frames for stabilization processing, and can be output as a frame of the recorded image. In other words, each frame includes a delay of x+y frames, and the image data captured by automatic tracking framing is output.
[0039] Note that while Figures 3 and 4 illustrate the case with x > y, x = y or x < y are also acceptable. Furthermore, although ±x frames are used as references for calculating image stabilization information, the number of past reference frames and future reference frames may be different. Similarly, although ±y frames are used as references for stabilization processing, the number of past reference frames and future reference frames may be different.
[0040] Here, we will explain how stabilizing the cropping frame 50 by referencing past and future frames as described above can result in a smooth tracking video of the subject. Figure 5 shows the original data JS as a thin line, the frame trajectory SP stabilized by referencing only past frames as a dashed line, and the frame trajectory SC stabilized by referencing both past and future frames as a thick line. The vertical axis represents a two-dimensional position in a one-dimensional way, and the horizontal axis represents time. The original data JS can be considered as the subject position in the captured image data. Note that subject recognition is lost in some sections.
[0041] When examining the frame trajectory SP stabilized by referencing only past frames, it appears that after the point of loss occurs, the only option is to maintain the extracted frame 50 from past frames. Furthermore, the frame position changes abruptly when subject recognition is restored. For this reason, the frame trajectory SP of the extracted frame 50 is not smooth.
[0042] On the other hand, when examining the frame trajectory SC stabilized by referencing past and future frames, smoothness is maintained even after the timing of the loss. This is because it is possible to reference future frames that have recovered from the loss, and therefore a smooth trajectory can be anticipated by expecting the extracted frame 50 to transition from the coordinates in the frame immediately before the loss to the coordinates in the recovery frame.
[0043] Next, Figure 6 explains image stabilization that references only past frames and image stabilization that references future frames. Figure 6 shows the results of applying "simple image stabilization" and "future-referenced image stabilization" to "camera shake." The vertical axis represents the position of the two-dimensional captured image in one dimension, and the horizontal axis represents time. In "camera shake," the range of the frame frame FW is shifted from frame to frame.
[0044] Simple camera shake correction refers to a case where in each frame, camera shake correction information is calculated without referencing at least future frames, and cropping is performed from the frame frame FW for camera shake correction. The cropping frame HW for camera shake correction is indicated by a broken line. That is, the frame frame FW corresponding to the pixel size of the image sensor is made larger than the pixel size of the frame frame to be output as an image, and each frame frame is cropped by the cropping frame HW in the direction of canceling blur, thereby performing camera shake correction. The trajectory KB is the trajectory of the center coordinates of the frame frame FW.
[0045] However, in this case, out-of-frame reference gr occurs as in the portion circled. Since there is no information of future frames, the out-of-frame reference gr cannot be predicted, and if the out-of-frame reference gr is abruptly avoided in a frame where the out-of-frame reference gr occurs, it will result in blur, which is not appropriate to avoid. This is due to the circumstances.
[0046] On the other hand, in the case of future reference type camera shake correction, the possibility of out-of-frame reference gr can be found from the camera shake information of future frames, for example, detection information of an IMU. Then, the trajectory KB of the frame frame to be cropped can be adjusted so as to avoid the out-of-frame reference gr from frames before the frame where the out-of-frame reference gr may occur. As a result, the frame trajectory of the cropping frame HW can be made a smooth trajectory while avoiding the out-of-frame reference gr as illustrated.
[0047] In the present embodiment, the concept of such future reference type camera shake correction is also applied to the stabilization of the cropping frame 50 for automatic tracking framing. In FIG. 7, "camera shake" is shown as in FIG. 6, and further "simple subject tracking" and "future reference type tracking" are shown. The vertical axis one-dimensionally represents the position of the two-dimensional captured image, and the horizontal axis represents time. A frame TW indicated by a broken line is defined as the range of the cropping frame 50 for subject tracking, that is, automatic tracking framing. The trajectory KT is the trajectory of the center coordinates of the cropping frame 50.
[0048] In simple subject tracking, the subject position is detected, and a cropping frame 50 centered on the subject in the composition is set for each frame. Assuming the subject remains at the same position, in the case of simple subject tracking, the trajectory KT becomes a straight line. However, in this case, an out-of-frame reference gr may occur, such as at a portion enclosed by a circle or ellipse. This is also because the out-of-frame reference gr cannot be predicted since there is no information on future frames.
[0049] On the other hand, in the case of future reference type tracking, the possibility of an out-of-frame reference gr can be found from camera shake information of future frames, for example, detection information from an IMU. Then, the trajectory KT of the cropping frame 50 can be adjusted from frames before the frame where the out-of-frame reference gr may occur, so as to avoid the out-of-frame reference gr. As a result, the frame trajectory of the frame TW can be made a smooth trajectory while avoiding the out-of-frame reference gr as illustrated.
[0050] <3. Processing Example> A specific example of the processing described above with reference to FIGS. 2 to 7 will be described with reference to FIG. 8. FIG. 8 is a flowchart of cropping processing for automatic tracking framing performed by the image processing unit 35.
[0051] Captured image data Idt from the imaging unit 31 is input to the image processing unit 35, and camera shake detection data Sd such as IMU data is input from the sensor unit 34. As the camera shake detection data Sd, for example, angular velocity information in three directions of yaw, pitch, and roll obtained by the IMU is used.
[0052] In FIG. 8, the output captured image data IdtP is a monitor image displayed on the display unit 32. The output captured image data IdtF is storage data stored as a moving image in the storage unit 36. That is, the processing of this flowchart is a processing example in which outputs for monitoring and for recording are performed by separate systems. The processing in FIG. 8 is performed for each frame of the captured image data Idt. Note that the buffer processing in steps S10, S11, S12, S20, S21, and S22 is processing in which the image processing unit 35 temporarily stores data in the buffer memory 35a (see FIG. 1).
[0053] First, let's explain the processing for monitoring. Here, we will explain the processing for frame FRc of the current captured image data Idt.
[0054] In step S1, the image processing unit 35 performs skeletal point recognition on the current frame FRc of the captured image data Idt, thereby recognizing a human subject within frame FRc. Then, in step S2, the image processing unit 35 generates a cropping frame 50 based on the subject recognition. That is, the image processing unit 35 sets the central coordinates and size of the cropping frame 50 so that the recognized subject is placed in an appropriate composition.
[0055] Meanwhile, in step S3, the image processing unit 35 calculates image stabilization information for the current frame FRc based on the image stabilization detection data Sd. In this case, the image stabilization information is calculated using information from a predetermined number of frames prior to the current frame FRc.
[0056] In step S4, the image processing unit 35 performs stabilization processing on the cropping frame 50. In this case, the image processing unit 35 performs image stabilization on the cropping frame 50 generated in step S2 using the image stabilization information generated in step S3. That is, it shifts the central coordinates of the cropping frame 50 according to the direction and amount indicated by the image stabilization information. Furthermore, it uses an IIR (Infinite Impulse Response) filter to stabilize the cropping frame 50, since it only uses information from past frames.
[0057] Then, in step S5, the image processing unit 35 performs image stabilization and cropping. In this case, the image processing unit 35 applies the image stabilization information generated in step S3 to the frame FRc of the captured image data Idt to correct the image stabilization of the image itself. Then, the image stabilization frame 50 stabilized in step S4 is applied to the image stabilization frame FRc to perform cropping.
[0058] This yields a single frame image extracted for tracking framing. The above processing is performed for each frame, and the captured image data IdtP is output to the display unit 32 as a monitor image. In this monitor image processing, there is no need to refer to information of future frames, so buffering is not required, and no buffering delay occurs.
[0059] Next, we will explain the memory processing. Here again, we will explain the processing from the perspective of frame FRc of the captured image data Idt.
[0060] Each frame of the captured image data Idt is buffered in step S11. The camera shake detection data Sd corresponding to the timing of each frame is buffered in step S12. The information of the cropped frame 50 generated in step S2 is buffered in step S10. The buffering in steps S10, S11, and S12 is a temporary storage for the x-frame period.
[0061] From the perspective of frame FRc, at the buffer timing of frame FDc+x, which is x frames later, information for future x frames becomes accessible. Therefore, in step S15, the image processing unit 35 generates image stabilization information for frame FRc. In this case, the image processing unit 35 generates image stabilization information by referring to past x frames and future x frames. As information for future x frames, the image stabilization detection data Sd from frame FDc+1 to frame FRc+x has been buffered in step S12, so this is referred to. As information for past frames, the image stabilization information for each frame from frame FDc-1 to frame FRc-x has already been calculated, so this is referred to.
[0062] The calculated image stabilization information is used in the stabilization process in step S30. In step S22, buffer processing is performed to temporarily store the image stabilization information for the y-frame period. In step S21, each frame of the captured image data Idt is also buffered to temporarily store for the y-frame period.
[0063] In step S30, the image processing unit 35 performs stabilization processing on the cropping frame 50 generated in step S2. That is, after buffer processing for x frames, the cropping frame 50 of frame FRc is targeted for processing.
[0064] In this case, the image processing unit 35 performs image stabilization on the cropping frame 50 generated in step S2 using the image stabilization information generated in step S15. That is, it shifts the central coordinate of the cropping frame 50 in the direction and by the amount indicated by the image stabilization information generated by referring to frames from the past to the future.
[0065] The information about this image-stabilized cropping frame (center coordinates and size) is buffered in step S20 as temporary storage for the required period. This allows future cropping frame information to be referenced during the stabilization process described above.
[0066] For example, from the perspective of frame FRc, at the timing after the y frame, the image stabilization cropping frame 50 for future y frames (from cropping frame CFc+1 to cropping frame CFc+y in Figure 4) becomes accessible. Information on the image stabilization cropping frame 50 for past y frames (from cropping frame CFc-1 to cropping frame CFc-y) has already been obtained. The image processing unit 35 performs stabilization processing on the cropping frame 50 of frame FRc by referencing these past and future ±y frame cropping frames 50.
[0067] The image processing unit 35 uses information from past to future frames and stabilizes the cropping frame 50 using an FIR (Finite Impulse Response) filter. Specifically, it smooths the central coordinates and frame size of the cropping frame 50. This smooths the frame trajectory. In this case, a Low Pass Filter (LPF) is configured with the FIR filter for stabilization. In particular, an averaging filter is configured in which the coefficients of all taps of the FIR filter are the same. Also, in this case, as described above, the system predicts the possibility of future loss periods and out-of-frame references and smooths to avoid them. That is, a smooth central coordinate trajectory is generated based on the past and future coordinates of the cropping frame 50 for each frame that has been stabilized. Even if there is a change in the size of the cropping frame 50, the system ensures that the change is smooth. In this process, the trajectory is generated by averaging the past and future coordinates, but lost frames are excluded from the calculation. Furthermore, the system also observes the future for out-of-frame references (gr), and when avoiding out-of-frame references (gr), it ensures a smooth frame trajectory without abrupt jumps in image placement.
[0068] Then, in step S40, the image processing unit 35 performs image stabilization and cropping. In this case, the image processing unit 35 applies the image stabilization information generated in step S15 to the frame FRc of the captured image data Idt to correct the image stabilization of the image itself. Then, the cropping process is performed by applying the cropping frame 50 stabilized in step S30 to the image stabilization frame FRc. This results in a cropped image of one frame as a tracking framing.
[0069] The above processing is performed for each frame, and the captured image data IdtF is output to the storage unit 36 as a stored image. In this processing of stored images, each frame is output with a delay of x + y frames.
[0070] <4. Summary and Modifications> The following effects can be obtained according to the above embodiments.
[0071] The imaging device 1 of this embodiment includes an image processing unit 35 that generates a cropping frame from the subject recognition result of the frame of the captured image data Idt, performs a stabilization process to stabilize the cropping frame based on information from multiple frames from past frames to future frames, and performs cropping processing on the frame of the captured image data Idt using the stabilized cropping frame. By stabilizing the cropping frame 50 for automatic tracking framing using not only past coordinates but also future coordinates, it is possible to set a cropping frame that captures the subject with smooth changes corresponding to future conditions and obtain stable tracking video. In particular, even if loss occurs repeatedly due to performance limitations of the subject recognition technology, it is possible to obtain stable and smooth tracking framing video. Furthermore, by observing the future, it is possible to prevent out-of-frame referencing, and smooth tracking can be achieved.
[0072] In the embodiment, the imaging device 1 generates image stabilization information using multiple frames from past to future for each frame of the captured image data Idt during the stabilization process. It generates a cropped frame from the subject recognition result of the frame, performs image stabilization on the cropped frame using the image stabilization information, and stabilizes the cropped frame based on the information from the past to the future of the image-stabilized cropped frame. This allows the cropped frame to be stabilized using image stabilization information generated from past to future frames, enabling the efficient execution of image stabilization and cropped frame stabilization while obtaining stable tracking video.
[0073] In this embodiment, the cropping frame 50 is defined by information regarding the frame size and a representative position (e.g., the center coordinates). This cropping frame is used to define the composition for capturing the subject.
[0074] In this embodiment, the image processing unit generates cropped image data by performing cropping on frames of captured image data that have been corrected for camera shake using image stabilization information, using a cropping frame that has undergone stabilization processing. In other words, the cropping frame 50 is stabilized after being corrected for camera shake based on past and future frames, the captured image data Idt is also corrected for camera shake based on past and future frames, and cropping is performed on the captured image data Idt using the stabilized cropping frame 50. This makes it possible to obtain a video with stable and smooth tracking framing.
[0075] In this embodiment, an imaging device 1 is described that includes a storage unit 36 for storing cropped image data generated by an image processing unit 35. In other words, the cropped imaging image data IdtF is stored in the storage unit 36, for example, as video data. When processing the stored data, image stabilization is performed using past and future information, and even when cropping is performed using a stabilized cropping frame 50, the user will not perceive any delay due to buffering.
[0076] In this embodiment, the image processing unit 35 calculates image stabilization information Hc using image stabilization detection data Sd for future frames obtained by buffering image stabilization detection data Sd corresponding to each frame of a predetermined number of frames (x frames) for the frame to be used for calculating image stabilization information, and image stabilization information Hc for past frames that has already been calculated (Hc-1 to Hc-x). This makes it possible to generate image stabilization information using past and future frames. In other words, image stabilization information based on information from past x frames and future x frames can be calculated as ±x frames. Finally, buffering of the captured image data Idt itself, which is the target of image stabilization, is only necessary for x + y frames.
[0077] In this embodiment, the image processing unit 35 buffers information of the cropped frame 50 corrected for camera shake using camera shake correction information, and performs stabilization processing using information of future cropped frames 50 and information of past cropped frames 50 for the frame to be calculated. This allows the cropped frame 50 to be stabilized using information of past and future cropped frames 50. Therefore, the state of the future cropped frame 50 can be predicted and the cropped frame 50 can be set to have an appropriate frame trajectory for each frame.
[0078] In this embodiment, the image processing unit 35 performs stabilization processing by excluding the information of the cropping frame for frames in which the subject was lost from the buffered information of future cropping frames 50 and past cropping frame 50. For frames in which the subject was lost, the cropping frame 50 from before the loss is maintained, for example, and therefore cannot be said to be an accurate cropping frame. For this reason, the information of the cropping frame 50 for lost frames is not used in the FIR filter calculation for smoothing. Then, by reflecting the cropping frame 50 of future frames in which the recognition of the subject is restored, a cropping frame 50 that will result in a smooth trajectory can be set.
[0079] In this embodiment, the image processing unit 35 performs a stabilization process that smooths the time-series trajectory of the representative position (central coordinate) of the cropping frame 50. This stabilizes the position of the cropping frame 50 so that it does not change abruptly, and a smooth tracking framing video can be obtained.
[0080] In this embodiment, the image processing unit 35 performs a stabilization process that smooths the time-series changes in the size of the cropping frame 50. This stabilizes the size of the cropping frame 50 so that it does not change abruptly, and a smooth tracking framing video can be obtained.
[0081] In this embodiment, the image processing unit performs a stabilization process by detecting frames in which an out-of-frame reference gr occurs within the cropping frame based on information about future cropping frames, and sets the cropping frame 50 for each frame so that a frame trajectory in which no out-of-frame reference gr occurs is obtained. This makes it possible to prevent image loss due to out-of-frame references during tracking framing.
[0082] In this embodiment, the image processing unit performs image stabilization on the frame of the captured image data Idt using image stabilization information generated using only past frames, and then performs cropping on the frame of the captured image data Idt using the cropping frame 50. In other words, the monitor image displayed on the display unit 32 is cropped using the cropping frame 50, which has been corrected using image stabilization information generated using only past frames. As a result, the displayed monitor image does not experience any delay due to referencing future information. In particular, even if momentary loss of subject recognition or out-of-frame referencing occurs during display, it does not pose a major problem, and conversely, the absence of delay results in a desirable display.
[0083] Although this explanation assumes video capture, the captured image data IDt may also be a preview image used for still image capture. Furthermore, the captured image data IDt may also be a still image. In addition, while automatic tracking and framing was explained using an example with a person as the subject, this technology can also be applied to subjects other than people, such as animals.
[0084] The program of this embodiment is a program that causes an information processing device such as a CPU or DSP, or a device including such a device, to perform a process like the one shown in Figure 8.
[0085] In other words, the program of this embodiment is a program that causes an image processing device to execute a stabilization process to stabilize the cropping frame based on the subject recognition result of the frame of the captured image data, and an extraction process to crop the frame of the captured image data using the stabilized cropping frame.
[0086] The functionality of this embodiment is realized when such a program is implemented in the image processing unit of a digital camera or included in, for example, a camera application on a smartphone.
[0087] Such programs can be pre-recorded on HDDs (hard disk drives) or ROMs (remote-controlled memory) within microcomputers, which are recording media built into computer devices. Alternatively, they can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed from removable recording media to personal computers, etc., or downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet.
[0088] Furthermore, such a program is suitable for providing the imaging device 1 of the embodiment to a wide range of users. For example, by downloading the program to mobile terminal devices such as smartphones and tablets, imaging devices, mobile phones, personal computers, game consoles, etc., these smartphones, etc., can be made to function as the imaging device 1 of this disclosure.
[0089] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.
[0090] Furthermore, this technology can also be configured as follows: (1) An imaging device comprising an image processing unit that generates a cropping frame from the subject recognition result of the frame of the captured image data, performs a stabilization process to stabilize the cropping frame based on information from multiple frames from past frames to future frames, and performs cropping processing on the frame of the captured image data using the cropping frame that has undergone the stabilization process. (2) The imaging device according to (1) above, wherein the image processing unit, in the stabilization process, generates image stabilization information for the frame of the captured image data using multiple frames from past frames to future frames, performs image stabilization on the cropping frame generated from the subject recognition result of the frame using the image stabilization information, and performs stabilization processing on the cropping frame based on information from the past to the future of the image stabilization process applied to the cropping frame. (3) The imaging device according to (1) or (2) above, wherein the cropping frame is set by information of the frame size and representative position. (4) The imaging device according to (2) above, wherein the image processing unit performs the cropping process on frames of image data that have been corrected for camera shake using the image stabilization information, using the cropping frame to which the stabilization process has been applied, in order to generate cropped image data. (5) The imaging device according to (4) above, further comprising a storage unit for storing cropped image data generated by the image processing unit. (6) The imaging device according to any one of (2), (4), or (5) above, wherein the image processing unit calculates camera shake correction information using camera shake detection data for future frames obtained by buffering camera shake detection data corresponding to each of a predetermined number of frames for the frames to be calculated for camera shake correction information, and camera shake correction information for past frames that has already been calculated. (7) The imaging device according to any one of (2), (4), (5), or (6) above, wherein the image processing unit buffers information of cropping frames corrected for camera shake using the image stabilization information, and performs the stabilization process using information of future cropping frames and information of past cropping frames for the frames to be calculated for cropping frames.(8) The imaging apparatus according to (7) above, wherein the image processing unit performs the stabilization process by excluding the information of the cropping frame of a frame in which the subject was lost from the buffered information of the future cropping frame and the information of the past cropping frame. (9) The imaging apparatus according to any one of (1) to (8) above, wherein the image processing unit performs the stabilization process by smoothing the trajectory of the time-series change of the representative position of the cropping frame. (10) The imaging apparatus according to any one of (1) to (9) above, wherein the image processing unit performs the stabilization process by smoothing the time-series change of the size of the cropping frame. (11) The imaging apparatus according to any one of (1) to (10) above, wherein the image processing unit detects frames in which out-of-frame references occur within the cropping frame from the information of the future cropping frame, and sets the cropping frame of each frame so that a change in the cropping frame in which no out-of-frame references occur can be obtained. (12) The imaging device according to any of (2), (4), (5), (6), (7), or (8) above, wherein the image processing unit performs image stabilization on the cropping frame using image stabilization information generated using only past frames from the current frame of the captured image data, as processing of the monitor display image, and performs cropping on the frame of the captured image data using the cropping frame. (13) An image processing method wherein the image processing device generates a cropping frame from the subject recognition result of the frame of the captured image data, performs stabilization processing to stabilize the cropping frame based on information from multiple frames from past frames to future frames, and performs cropping on the frame of the captured image data using the cropping frame after the stabilization processing. (14) A program that causes the image processing device to execute stabilization processing to generate a cropping frame from the subject recognition result of the frame of the captured image data, stabilization processing to stabilize the cropping frame based on information from multiple frames from past frames to future frames, and cropping processing to crop the frame of the captured image data using the cropping frame after the stabilization processing.
[0091] 1. Imaging device 30. Control unit 31. Imaging unit 32. Display unit 33. Operation unit 34. Sensor unit 35. Image processing unit 36. Storage unit 37. Memory unit 50. Cropping frame
Claims
1. An imaging device comprising an image processing unit that generates a cropping frame from the subject recognition result of the frame of the captured image data, performs a stabilization process to stabilize the cropping frame based on information from multiple frames from past frames to future frames, and performs cropping processing on the frame of the captured image data using the cropping frame that has undergone the stabilization process.
2. The imaging device according to claim 1, wherein the image processing unit, in the stabilization process, generates image stabilization information for a frame of captured image data using multiple frames from past frames to future frames, performs image stabilization on a cropped frame generated from the subject recognition result of the frame using the image stabilization information, and stabilizes the cropped frame based on the information from the past to the future of the cropped frame to which image stabilization has been applied.
3. The imaging apparatus according to claim 1, wherein the cropping frame is set by information of the frame size and representative position.
4. The imaging apparatus according to claim 2, wherein the image processing unit, in the cropping process, performs a cropping process on the frame of the image image data that has been corrected for camera shake using the camera shake correction information, using the cropping frame that has been stabilized, to generate cropped image data.
5. The imaging apparatus according to claim 4, further comprising a storage unit for storing cropped image data generated by the image processing unit.
6. The imaging device according to claim 2, wherein the image processing unit calculates image stabilization information using image stabilization detection data for future frames obtained by buffering image stabilization detection data corresponding to each of a predetermined number of frames for which image stabilization information is to be calculated, and image stabilization information for past frames that has already been calculated.
7. The imaging apparatus according to claim 2, wherein the image processing unit buffers information of the cropped frame corrected for camera shake using the camera shake correction information, and performs the stabilization process using the information of the future cropped frame and the information of the past cropped frame for the frame to be calculated for the cropped frame.
8. The imaging apparatus according to claim 7, wherein the image processing unit performs the stabilization process by excluding the information of the cropping frame in which the subject was lost from the buffered information of the future cropping frame and the information of the past cropping frame.
9. The imaging apparatus according to claim 1, wherein the image processing unit performs a process to smooth the trajectory of the time-series change of the representative position of the cropping frame as the stabilization process.
10. The imaging apparatus according to claim 1, wherein the image processing unit performs a process to smooth the time-series change in the size of the cropping frame as the stabilization process.
11. The imaging apparatus according to claim 1, wherein the image processing unit, as the stabilization process, detects frames in which out-of-frame references occur within the cropping frame from information on future cropping frames, and sets the cropping frame for each frame so that changes in the cropping frame that do not result in out-of-frame references are obtained.
12. The imaging apparatus according to claim 2, wherein the image processing unit performs image stabilization on the cropping frame using image stabilization information generated using only past frames from the current frame of the captured image data, and performs cropping on the frame of the captured image data using the cropping frame.
13. An image processing method comprising: an image processing device generating a cropping frame from the subject recognition result of a frame of captured image data; performing a stabilization process to stabilize the cropping frame based on information from multiple frames from past frames to future frames; and performing a cropping process on a frame of captured image data using the cropping frame that has undergone the stabilization process.
14. A program that causes an image processing device to perform a stabilization process to generate a cropping frame from the subject recognition results of the captured image data frame, stabilize the cropping frame based on information from multiple frames from past frames to future frames, and crop the captured image data frame using the cropping frame that has undergone the stabilization process.
