Color correction method and program

Abstract

A color correction device: performs first color correction on images of respective frames of a video captured by an imaging device, the first color correction being based on an image captured at a reference time; performs second color correction, of higher performance than the first color correction, when state information including at least one among the position and orientation of the imaging device has changed by at least a predetermined value from state information at the reference time, and sets a first time, at which the second color correction was performed, as a new reference time; outputs the images color-corrected by the second color correction, at the reference time; and outputs the images color-correction by the first color correction, at a time other than the reference time.

Description

Color correction method and program

[0001] The disclosed technology relates to a color correction method and a color correction program.

[0002] When shooting underwater, the amount of light absorbed varies greatly depending on the wavelength of light, resulting in a color cast or other color distortion in the captured image.

[0003] As a technique to suppress color cast, for example, an imaging device has been proposed that converts incident subject light into an electrical signal to generate an image signal, and adjusts the signal level of each color component using gains set for each color component of the image signal. This imaging device generates discrimination information to determine position changes. This imaging device also has a translucent diffuser plate that can be inserted and removed in the optical path of the subject light incident on the imaging unit. When the operating mode is set to underwater imaging mode, if the position change determined based on the discrimination information exceeds a threshold, the diffuser plate is inserted into the optical path, and the gain for each color component is set using the image signal generated by the imaging unit at that time.

[0004] Japanese Patent Publication No. 2009-159469

[0005] However, the conventional technology described above has the problem that it is necessary to implement a diffuser plate in the imaging device. Furthermore, with the conventional technology described above, the diffuser plate is photographed when the gain is set, so the image of the subject cannot be used as the image, and it is difficult to output a color-corrected image in real time.

[0006] Furthermore, recent advancements in deep learning technology have made high-performance color correction possible from a single still image. While it is conceivable to use this deep learning technology for color correction instead of setting the gain using a diffuser plate, there is a problem in that the computational cost of color correction using deep learning technology is very high.

[0007] One aspect of the disclosure technology is that it aims to reduce the computational cost of image color correction and ensure real-time performance.

[0008] In one embodiment, the disclosed technology performs a first color correction on each frame of video captured by the camera, based on an image captured at a reference time. Furthermore, if state information including at least one of the position and orientation of the camera changes by a predetermined value or more from the state information at the reference time, the disclosed technology performs a second color correction that is more efficient than the first color correction. The disclosed technology also updates the first time at which the second color correction was performed to a new reference time. Then, at the reference time, the disclosed technology outputs the image after color correction by the second color correction, and at times other than the reference time, it outputs the image after color correction by the first color correction.

[0009] One aspect of this approach is that it reduces the computational cost of image color correction, thereby ensuring real-time performance.

[0010] This is a block diagram illustrating the schematic configuration of an image processing system according to the first to fourth embodiments. This is a functional block diagram of a color correction device according to the first embodiment. This is a diagram illustrating simple color correction. This is a diagram illustrating high-performance color correction. This is a diagram illustrating the time at which high-performance color correction and simple color correction are performed. This is a block diagram illustrating the schematic configuration of a computer functioning as a color correction device according to the first to fourth embodiments. This is a flowchart illustrating an example of color correction processing according to the first embodiment. This is a functional block diagram of a color correction device according to the second embodiment. This is a functional block diagram of a color correction device according to the third embodiment. This is a flowchart illustrating an example of color correction processing according to the third embodiment. This is a functional block diagram of a color correction device according to the fourth embodiment. This is a flowchart illustrating an example of color correction processing according to the fourth embodiment.

[0011] An example of an embodiment relating to the disclosed technology will be described below with reference to the drawings.

[0012] <First Embodiment> Figure 1 is a block diagram showing the schematic configuration of the image processing system 100 according to the first embodiment. As shown in Figure 1, the image processing system 100 includes a color correction device 10 and an underwater photography device 30. The color correction device 10 and the underwater photography device 30 are connected via a network.

[0013] The underwater photography device 30 includes a moving mechanism 32, a photography device 34, and a sensor 36.

[0014] The moving mechanism 32 is a mechanism for moving the underwater camera 30 forward, backward, rotated, and in the depth direction underwater. The moving mechanism 32 may be, for example, an underwater drone. The moving mechanism 32 moves the underwater camera 30 underwater in response to control from a controller (not shown) or according to a preset program.

[0015] The imaging device 34 may be a visible light camera capable of capturing images underwater. The imaging device 34 outputs the captured images to the color correction device 10.

[0016] Sensor 36 is a group of sensors for detecting the position (X and Y coordinates) and orientation (roll angle, pitch angle, yaw angle) of the imaging device 34, and includes, for example, a gyro sensor, an accelerometer, a depth sensor, etc. Sensor 36 is attached to the imaging device 34 or the moving mechanism 32 on which the imaging device 34 is mounted. Sensor 36 outputs data indicating the detected position and orientation of the imaging device 34 (hereinafter referred to as "position and orientation information") to the color correction device 10. Note that the position and orientation information is an example of "state information" in the disclosed technology.

[0017] It is assumed that the time of capture of each frame of the video captured by the camera 34 corresponds to the time of detection of position and orientation information by the sensor 36. In the following, "time" refers to the time of capture of each frame and the corresponding time of detection of position and orientation information.

[0018] Figure 2 is a functional block diagram of the color correction device 10. As shown in Figure 2, the color correction device 10 functionally includes a first color correction unit 12, a second color correction unit 14, and a determination unit 16.

[0019] The first color correction unit 12 performs a simplified color correction on each frame of the video captured by the shooting device 34, based on the image captured at a reference time, and outputs the color-corrected image. Specifically, the first color correction unit 12 converts the image before color correction to the color-corrected image using a correction coefficient calculated from the image before and after color correction by the second color correction unit 14, which will be described later.

[0020] More specifically, the first color correction unit 12 acquires pairs of images taken at a reference time, one before color correction and one after color correction, from the second color correction unit 14. The "reference time" will be explained later. The first color correction unit 12 calculates a correction coefficient for each RGB component, such that the corrected image = correction coefficient × the image before color correction. Then, as shown in Figure 3, the first color correction unit 12 takes the image before color correction of each frame of the video input from the shooting device 34 as the input image, and multiplies each RGB component of the input image by the calculated correction coefficient. The first color correction unit 12 synthesizes the images of each component after multiplication by the correction coefficient to generate a corrected image, which is then output as the output image.

[0021] The second color correction unit 14 performs high-performance color correction at a reference time. High-performance color correction is a color correction that is more advanced than the color correction performed by the first color correction unit 12. Specifically, the second color correction unit 14 takes the image captured by the shooting device 34 before color correction as input to a program that performs color correction on the input image, and then acquires and outputs the image after color correction.

[0022] The program that performs color correction may be, for example, a deep learning model such as RAUNE-Net that has been trained for color correction. As shown in Figure 4, the machine learning model is trained so that when an uncolor-corrected image is input, it outputs an image similar to the corresponding color-corrected image, by pairing a set of uncolor-corrected images taken underwater with a set of corresponding color-corrected images. The color-corrected images used to train the machine learning model may be images that have been manually color-corrected, or images in which color cast has been eliminated using lighting during shooting.

[0023] The determination unit 16 calculates an evaluation value for each time point to evaluate the amount of change in the position and orientation of the imaging device 34, based on the position and orientation information acquired from the sensor 36. The evaluation value may be any of the X coordinate, Y coordinate, depth, roll angle, pitch angle, and yaw angle, or it may be a combined value of two or more of these. The evaluation value only needs to evaluate the change in at least one of the position and orientation of the imaging device 34. However, since the attenuation conditions that affect the color cast of the image vary not only by the water depth but also by the direction of shooting, the distance to the subject, the positional relationship with the sun, etc., it is desirable to calculate an evaluation value that can be comprehensively determined by combining various factors.

[0024] The determination unit 16 determines whether the amount of change in the calculated evaluation value from the reference point is greater than or equal to a predetermined threshold. If the determination unit 16 determines that the amount of change in the evaluation value is greater than or equal to the threshold, it causes the second color correction unit 14 to perform high-performance color correction and updates the time at which the high-performance color correction was performed to a new reference time. Furthermore, if the reference time is updated, the determination unit 16 instructs the first color correction unit 12 to calculate a correction coefficient based on the new reference time and causes it to perform simplified color correction using the newly calculated correction coefficient.

[0025] If color correction using a fixed correction coefficient is applied to all frames of images to reduce computational costs, changes in the position and orientation of the imaging device 34 may prevent the system from following changes in the attenuation conditions that cause color cast. In this case, accurate color correction cannot be performed. On the other hand, if high-performance color correction is applied to all frames of images to achieve accurate color correction, the computational costs become extremely high.

[0026] In the color correction device 10 according to this embodiment, as shown in Figure 5, the time when high-performance color correction is performed by the second color correction unit 14 is set as the reference time, and between the reference time and the next reference time (for example, A in Figure 5), simplified color correction is performed by the first color correction unit 12. In other words, high-performance color correction, which has a high computational cost, is performed only when necessary, and simplified color correction, which has a low computational cost, is performed during other periods. Therefore, the computational cost of color correction is suppressed. In addition, since the color-corrected image of the captured image is output in every frame, real-time performance is also ensured.

[0027] The color correction device 10 may be implemented, for example, by a computer 40 as shown in Figure 6. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes input / output devices 45 such as input devices and display devices, and an R / W (Read / Write) device 46 that controls the reading and writing of data to and from the storage medium 49. The computer 40 also includes a communication interface 47 that connects to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output devices 45, R / W device 46, and communication interface 47 are connected to each other via a bus 48.

[0028] The storage device 44 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. The storage device 44, as a storage medium, stores a color correction program 50 that causes the computer 40 to function as a color correction device 10. The color correction program 50 includes a first color correction process control command 52, a second color correction process control command 54, and a determination process control command 56.

[0029] The CPU 41 reads the color correction program 50 from the storage device 44, loads it into memory 43, and sequentially executes the control instructions contained in the color correction program 50. The CPU 41 operates as the first color correction unit 12 shown in Figure 2 by executing the first color correction process control instruction 52. The CPU 41 also operates as the second color correction unit 14 shown in Figure 2 by executing the second color correction process control instruction 54. Furthermore, the CPU 41 operates as the determination unit 16 shown in Figure 2 by executing the determination process control instruction 56. As a result, the computer 40 that executed the color correction program 50 functions as a color correction device 10. The CPU 41 that executes the program is hardware. Also, part of the program may be executed by the GPU 42.

[0030] The functions realized by the color correction program 50 may be implemented, for example, by a semiconductor integrated circuit, more specifically by an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc.

[0031] Next, the operation of the image processing system 100 according to the first embodiment will be described. The underwater shooting device 30 moves underwater by the moving mechanism 32 and starts shooting with the shooting device 34. Then, when the video captured by the shooting device 34 and the position and orientation information detected by the sensor 36 are sequentially input to the color correction device 10, the color correction device 10 performs the color correction process shown in Figure 7. Note that the color correction process is an example of a color correction method of the disclosed technology.

[0032] In step S10, the first color correction unit 12 and the second color correction unit 14 acquire an image of one frame of the video input to the color correction device 10 as an input image. The determination unit 16 also acquires position and orientation information input to the color correction device 10.

[0033] Next, in step S12, the determination unit 16 calculates an evaluation value for evaluating the amount of change in the position and orientation of the imaging device 34 from the reference time on the basis of the position and orientation information acquired from the sensor 36. Then, the determination unit 16 determines whether or not the amount of change in the calculated evaluation value from the reference time is equal to or greater than a predetermined threshold. If the amount of change in the evaluation value is equal to or greater than the threshold, the process proceeds to step S14, and if the amount of change is less than the threshold, the process proceeds to step S20.

[0034] In step S14, the second color correction unit 14 inputs the input image acquired in step S10, that is, the image before color correction, into a machine learning model trained to perform color correction on an input image. Then, the second color correction unit 14 acquires, from the machine learning model, the image after color correction on which high-performance color correction has been performed, and outputs the acquired image as an output image.

[0035] Next, in step S16, the first color correction unit 12 acquires a pair of the image before color correction and the image after color correction on which high-performance color correction was performed in step S14. Then, the first color correction unit 12 calculates, for each RGB component, a correction coefficient that satisfies image after color correction = correction coefficient × image before color correction.

[0036] Next, in step S18, the determination unit 16 updates the time at which high-performance color correction was performed by the second color correction unit 14 in step S14 to a new reference time, and stores the position and orientation information at the new reference time in a predetermined storage area.

[0037] On the other hand, in step S20, the first color correction unit 12 performs simple color correction by multiplying the input image before color correction by the correction coefficient calculated in the most recent step S16, converts the input image before color correction into an output image after color correction, and outputs the output image.

[0038] Next, in step S22, the determination unit 16 determines whether or not to end the color correction process by determining whether video input has ended, whether a command instructing to end the color correction process has been received, or the like. If the color correction process is not to be ended, the process returns to step S10, and the process is continued for the image of the next frame. If the color correction process is not to be continued, the color correction process shown in FIG. 7 is ended.

[0039] As described above, according to the image processing system of the first embodiment, the color correction device performs a first color correction (simple color correction) on each frame of the video captured by the shooting device, based on the image captured at a reference time. Furthermore, if the position and orientation information of the shooting device changes by more than a threshold from the position and orientation information at the reference time, the color correction device performs a high-performance color correction that is more efficient than the simple color correction, and updates the time at which the high-performance color correction was performed to a new reference time. The color correction device then outputs the image after color correction by the second color correction at the reference time, and outputs the image after color correction by the first color correction at times other than the reference time. This reduces the computational cost of color correction of images and ensures real-time performance.

[0040] <Second Embodiment> Next, a second embodiment will be described. In the second embodiment, components with the same reference numerals as those in the image processing system 100 according to the first embodiment will be used, and detailed descriptions will be omitted. Also, for functional components that share some functions with the functional components of the first embodiment, the last two digits of the reference numerals will be the same, and detailed descriptions will be omitted.

[0041] As shown in Figure 1, the image processing system 200 according to the second embodiment includes a color correction device 210 and an underwater imaging device 30. The color correction device 210 and the underwater imaging device 30 are connected via a network.

[0042] As shown in Figure 8, the color correction device 210 functionally includes a first color correction unit 12, a second color correction unit 14, and a determination unit 216.

[0043] In the first embodiment, the determination unit 16 determined whether the amount of change in the evaluation value calculated from the position and orientation information of the imaging device 34 is greater than or equal to a threshold, based on the position and orientation information detected by the sensor 36. In the second embodiment, the determination unit 216 calculates the evaluation value based on the input image.

[0044] Specifically, the determination unit 216 calculates an evaluation value based on the change in the distribution of each RGB component in the input image. For example, the determination unit 216 may calculate the evaluation value as the sum of the differences in the values ​​of each pixel of each RGB component in the current input image and the input image at the reference time. Also, if the imaging device 34 is capable of measuring depth, the determination unit 216 may calculate the evaluation value based on the change in the distribution of depth information for each pixel of the input image. When the determination unit 216 calculates the evaluation value from the changes in the values ​​of the RGB components and the distribution of depth information, it may use the distribution near the center of the input image.

[0045] The color correction device 210 may be implemented, for example, by a computer 40 as shown in Figure 6. The storage device 44 of the computer 40 stores a color correction program 250 for causing the computer 40 to function as a color correction device 210. The color correction program 250 includes a first color correction process control command 52, a second color correction process control command 54, and a determination process control command 256.

[0046] The CPU 41 reads the color correction program 250 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions contained in the color correction program 250. By executing the determination process control instruction 256, the CPU 41 operates as the determination unit 216 shown in Figure 8. The other control instructions are the same as those for the color correction program 50 according to the first embodiment. As a result, the computer 40 that executed the color correction program 250 functions as a color correction device 210. The CPU 41 that executes the program is hardware. Also, part of the program may be executed by the GPU 42.

[0047] The functions realized by the color correction program 250 may be implemented, for example, by a semiconductor integrated circuit, more specifically by an ASIC, FPGA, etc.

[0048] The operation of the image processing system 200 according to the second embodiment is the same as in the first embodiment, except that in step S12 of the color correction process shown in Figure 7, an evaluation value is calculated from the input image instead of using the position and orientation information detected by the sensor 36. Therefore, a description will be omitted.

[0049] As described above, according to the image processing system of the second embodiment, the color correction device calculates an evaluation value from the input image to evaluate the amount of change in the position and orientation information of the imaging device. This makes it possible to determine the time to perform high-performance color correction without acquiring position and orientation information from the sensor.

[0050] <Third Embodiment> Next, a third embodiment will be described. In the third embodiment, components with the same reference numerals as those in the image processing system 100 according to the first embodiment will be used, and detailed descriptions will be omitted. Also, for functional components that share some functions with the functional components of the first embodiment, the last two digits of the reference numerals will be the same, and detailed descriptions will be omitted.

[0051] As shown in Figure 1, the image processing system 300 according to the third embodiment includes a color correction device 310 and an underwater imaging device 30. The color correction device 310 and the underwater imaging device 30 are connected via a network.

[0052] As shown in Figure 9, the color correction device 310 functionally includes a first color correction unit 12, a second color correction unit 14, and a prediction unit 318.

[0053] The prediction unit 318 predicts the time when the change in an evaluation value for evaluating the amount of change in position and orientation information from the reference time exceeds a threshold, based on position and orientation information at multiple time points after the reference time. This time is an example of the "second time" of the disclosed technology. For example, the prediction unit 318 may calculate evaluation values ​​at multiple time points after the reference time and predict future evaluation values ​​by extrapolation from the calculated evaluation values.

[0054] The prediction unit 318 instructs the second color correction unit 14 to perform high-performance color correction at a predetermined time before the time at which it predicts the change in the evaluation value will exceed a threshold. This predetermined time is set in advance to take into account the time required to calculate the correction coefficient. Furthermore, once the high-performance color correction by the second color correction unit 14 is completed, the prediction unit 318 instructs the first color correction unit 12 to start calculating the correction coefficient and to perform simplified color correction using the calculated correction coefficient from the predicted time.

[0055] The color correction device 310 may be implemented, for example, by a computer 40 as shown in Figure 6. The storage device 44 of the computer 40 stores a color correction program 350 for causing the computer 40 to function as a color correction device 310. The color correction program 350 includes a first color correction process control command 52, a second color correction process control command 54, and a prediction process control command 358.

[0056] The CPU 41 reads the color correction program 350 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions contained in the color correction program 350. By executing the prediction process control instruction 358, the CPU 41 operates as the prediction unit 318 shown in Figure 9. The other control instructions are the same as those for the color correction program 50 according to the first embodiment. As a result, the computer 40 that executes the color correction program 350 functions as a color correction device 310. The CPU 41 that executes the program is hardware. Also, part of the program may be executed by the GPU 42.

[0057] The functions realized by the color correction program 350 may be implemented, for example, by a semiconductor integrated circuit, more specifically by an ASIC, FPGA, etc.

[0058] Next, the operation of the image processing system 300 according to the third embodiment will be described. The underwater shooting device 30 moves underwater by the moving mechanism 32 and starts shooting with the shooting device 34. Then, when the video captured by the shooting device 34 and the position and orientation information detected by the sensor 36 are sequentially input to the color correction device 310, the color correction device 310 executes the color correction process shown in Figure 10. Note that the color correction process is an example of a color correction method of the disclosed technology. In the color correction process shown in Figure 10, processes that are the same as the color correction process in the first embodiment (Figure 7) are given the same step numbers and detailed explanations are omitted.

[0059] Following step S10, in step S310, the prediction unit 318 predicts the time when the change in the evaluation value for evaluating the change in position and orientation information from the reference time exceeds a threshold, based on position and orientation information at multiple time points after the reference time.

[0060] Next, in step S312, the prediction unit 318 determines whether or not it is a predetermined time before the time predicted in step S310. If it is a predetermined time before the predicted time, the process proceeds to steps S14 to S18; otherwise, the process proceeds to step S314.

[0061] In step S314, the prediction unit 318 determines whether the time is as predicted in step S310. If it is the predicted time, the process proceeds to step S316, where the first color correction unit 12 updates the correction coefficient used for simplified color correction to the correction coefficient calculated in the most recent step S16, and proceeds to step S20. If it is not the predicted time, step S316 is skipped, and the process proceeds to step S20. Then, after steps S20 and S22, the color correction process is completed.

[0062] As described above, according to the image processing system of the third embodiment, the color correction device predicts the time when the amount of change in the evaluation value for evaluating the amount of change in the position and orientation information of the imaging device exceeds a threshold. The color correction device then starts calculating the correction coefficients used in simplified color correction a predetermined time before the predicted time so that the correction coefficients used in simplified color correction are updated at the predicted time. This prevents a delay in updating the correction coefficients by the time required to calculate them, and allows the correction coefficients in simplified color correction to be updated at an appropriate time.

[0063] <Fourth Embodiment> Next, a fourth embodiment will be described. In the fourth embodiment, components with the same reference numerals as those in the image processing system 100 according to the first embodiment will be used, and detailed descriptions will be omitted. Also, for functional components that share some functions with the functional components of the first embodiment, the last two digits of the reference numerals will be the same, and detailed descriptions will be omitted.

[0064] As shown in Figure 1, the image processing system 400 according to the fourth embodiment includes a color correction device 410 and an underwater imaging device 30. The color correction device 410 and the underwater imaging device 30 are connected via a network.

[0065] In the first to third embodiments, the case where video captured by the shooting device 34 is sequentially input to the color correction device via a network, i.e., online processing, was described. In the fourth embodiment, the video captured by the shooting device 34 for a predetermined time and the position and orientation information detected by the sensor 36 for a predetermined time are temporarily stored in a database. Then, the image of each frame is read from the database and color correction is performed, in a case of so-called offline processing, which will be described.

[0066] As shown in Figure 11, the color correction device 410 functionally includes a first color correction unit 412, a second color correction unit 414, and a determination unit 416. Video footage captured by the imaging device 34 for a predetermined period of time is stored in the video database, and position and orientation information detected by the sensor 36 for a predetermined period of time is stored in the position and orientation database.

[0067] The determination unit 416 acquires position and orientation information from the position and orientation DB and calculates an evaluation value to evaluate the amount of change in position and orientation information from a reference time at each time point. The determination unit 416 identifies the time points in order from the start time when the amount of change in the evaluation value exceeds a threshold, sets the identified time points as new reference times, and repeats the process of identifying the time points when the amount of change in the evaluation value exceeds a threshold. In this way, the determination unit 416 identifies all reference times within a predetermined period, i.e., the time points at which high-performance color correction is performed by the second color correction unit 414.

[0068] The second color correction unit 414 sequentially acquires images from the video database starting from the frame at the start time, performs high-performance color correction at the time identified by the determination unit 416, and outputs the color-corrected output image.

[0069] The first color correction unit 412 interpolates the correction coefficient for the simplified color correction performed between time T1, when high-performance color correction is performed by the second color correction unit 414, and time T2, when high-performance color correction is performed next, based on the correction coefficients at times T1 and T2. Specifically, the first color correction unit 412 interpolates the correction coefficient at each time between time T1 and time T2 so that it gradually changes from the correction coefficient at time T1 to the correction coefficient at time T2. Time T1 is an example of the "third time" of the disclosed technology, and time T2 is an example of the "fourth time" of the disclosed technology.

[0070] The interpolation method for the correction coefficient may be linear interpolation, spline interpolation, or the like. The first color correction unit 412 may also input the shooting environment conditions and the correction coefficient at time T1 into the prediction model to calculate the correction coefficient between time T1 and time T2. The shooting environment conditions are factors that affect the attenuation conditions underwater, such as changes in the position and orientation of the shooting device 34, time, season, and weather. Various models can be applied as the prediction model, such as models based on physical laws and models based on time series data. For example, a Kalman filter, particle filter, or recurrent neural network may be applied as the prediction model.

[0071] Furthermore, the correction coefficients input to the prediction model may be in quaternion format. For example, the first color correction unit 412 may use a state transition model as the prediction model and input a state vector including a quaternion representing the direction vector and angle using the correction coefficients to the prediction model. This case will be explained in more detail.

[0072] The first color correction unit 412 sets the correction coefficient for each RGB color component to K = (k Ｒ ,k Ｇ ,k Ｂ When this is the case, the correction coefficient K is a quaternion Q where the rotation angle is norm(K) * γ (where γ is an arbitrary coefficient) and the direction vector is K / norm(K). Ｌ Convert to.

[0073] Here, as a predictive model, we will explain using a so-called state equation, which predicts the next state after one sample from the current state. The state vector is x = [Pc Qc Vc Wc T Q]Ｌ Ｔ Let Pc=(px, py, pz), Qc=(q0, q1, q2, q3), Vc=(vx, vy, vz), and Wc=(wx, wy, wz) respectively represent the position, posture, velocity, and angular velocity of the imaging device 34. T represents date and time, and is a factor for considering the change in solar altitude with time.

[0074] The state transition equation is defined as x ｋ＋１ =f(x ｋ , θ)+w ｋ where f(x ｋ , θ) is a nonlinear state transition function, θ is a parameter of the dynamic model of Q Ｌ , and w ｋ represents process noise. The observation equation is defined as z ｋ =h(x ｋ )+v ｋ where h(x ｋ ) is an observation function, and v ｋ represents observation noise.

[0075] The first color correction unit 412 predicts correction coefficients and updates a prediction model by executing the following steps. Initialization step: set the parameter θ of the prediction model to an arbitrary initial value. Prediction step: predict the state x ｋ and the covariance matrix using an Extended Kalman Filter (EKF) or an Unscented Kalman Filter (UKF). Update step: when a new observation z ｋ is obtained, update the state and the covariance matrix using EKF or UKF. At the same time, update the parameter θ of the dynamic model of Q Ｌ using the RLS or EM algorithm. Repeat step: repeat the prediction step and the update step until a termination condition is satisfied.

[0076] Being three parameters, which correspond to correction coefficients for each of the RGB color components, has good compatibility with quaternions representing three-dimensional postures (angles), and enables easy smooth interpolation.

[0077] The first color correction unit 412 performs a simplified color correction on the image of each frame between times T1 and T2 using the correction coefficients for each time calculated by interpolation.

[0078] The color correction device 410 may be implemented, for example, by a computer 40 as shown in Figure 6. The storage device 44 of the computer 40 stores a color correction program 450 for causing the computer 40 to function as a color correction device 410. The color correction program 450 includes a first color correction process control command 452, a second color correction process control command 454, and a determination process control command 456.

[0079] The CPU 41 reads the color correction program 450 from the storage device 44, loads it into memory 43, and sequentially executes the control instructions contained in the color correction program 450. By executing the first color correction process control instruction 452, the CPU 41 operates as the first color correction unit 412 shown in Figure 11. Furthermore, by executing the second color correction process control instruction 454, the CPU 41 operates as the second color correction unit 414 shown in Figure 11. Furthermore, by executing the determination process control instruction 456, the CPU 41 operates as the determination unit 416 shown in Figure 11. As a result, the computer 40 that executed the color correction program 450 functions as a color correction device 410. Note that the CPU 41 that executes the program is hardware. Also, part of the program may be executed by the GPU 42.

[0080] The functions realized by the color correction program 450 may be implemented, for example, by a semiconductor integrated circuit, more specifically by an ASIC, FPGA, etc.

[0081] Next, the operation of the image processing system 400 according to the fourth embodiment will be described. The underwater shooting device 30 moves underwater by the movement mechanism 32, and the video captured by the shooting device 34 for a predetermined period of time is stored in the video DB. In addition, position and orientation information for a predetermined period of time detected by the sensor 36 is stored in the position and orientation DB. Then, the color correction device 410 performs the color correction process shown in Figure 12. Note that the color correction process is an example of the color correction method of the disclosed technology. Furthermore, in the color correction process shown in Figure 12, the same process as the color correction process in the first embodiment (Figure 7) is given the same step number and detailed explanation is omitted.

[0082] In step S410, the determination unit 416 acquires position and orientation information from the position and orientation DB and calculates evaluation values ​​to evaluate the amount of change in position and orientation information from the reference time at each time, starting from the start time. The determination unit 416 then identifies times T1 and T2 in which the amount of change in the evaluation value is equal to or greater than a threshold.

[0083] Next, in step S412, the second color correction unit 414 sequentially acquires images from the video DB, starting with the frame at the start time, and performs high-performance color correction at the times T1 and T2 identified in step S410.

[0084] Next, in step S414, the first color correction unit 412 interpolates the correction coefficient so that it gradually changes from the correction coefficient at time T1 to the correction coefficient at time T2 at each time between time T1 and time T2. As a result, the first color correction unit 412 calculates a correction coefficient for simplified color correction between time T1 and T2.

[0085] Next, in step S416, the first color correction unit 412 performs a simplified color correction on the image of each frame between times T1 and T2, using the correction coefficients for each time calculated by interpolation in step S414.

[0086] Next, in step S418, the first color correction unit 412 and the second color correction unit 414 output the color-corrected output images for each frame from time T1 to T2 in chronological order. If it is determined in step S22 to continue the color correction process, the process returns to step S410, sets the currently identified time T2 to a new time T1, and repeats the process of identifying the next time T2.

[0087] As described above, according to the image processing system of the fourth embodiment, the color correction device calculates a correction coefficient to be used for simplified color correction during offline processing by interpolation between the time periods in which high-performance color correction is performed. This makes it possible to achieve smooth color correction that suppresses abrupt changes in color.

[0088] In the fourth embodiment, if the difference between the correction coefficient at time T1 and the correction coefficient at time T2 is greater than or equal to a predetermined value, a reference time may be added between times T1 and T2, and high-performance color correction may also be performed on the added reference time. This makes it possible to achieve smoother color correction.

[0089] Furthermore, while the above embodiments describe the calculation of evaluation values ​​for evaluating the amount of change in the position and orientation information of the imaging device, the system is not limited to these. A threshold may be set for each piece of information included in the position and orientation information, and when the amount of change in any of these pieces of information exceeds the threshold, it may be determined that it is time to perform high-performance color correction. For example, it may be determined that the water depth has changed by more than a threshold relative to a reference time, that the horizontal movement has changed by more than a threshold, or that the line of sight has changed by more than a threshold.

[0090] Furthermore, in each of the above embodiments, the color correction program is pre-stored (installed) in the storage device, but is not limited thereto. The program relating to the disclosed technology may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, or USB memory.

[0091] 10, 210, 310, 410 Color correction device 12, 412 First color correction unit 14, 414 Second color correction unit 16, 216, 416 Judgment unit 318 Prediction unit 30 Underwater photography device 32 Moving mechanism 34 Photography device 36 Sensor 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50, 250, 350, 450 Color correction program 52, 452 First color correction process control command 54, 454 Second color correction process control command 56, 256, 456 Judgment process control command 358 Prediction process control command 100, 200, 300, 400 Image processing system

Claims

1. A color correction method performed by a computer, which includes: performing a first color correction on each frame of video captured by a camera based on an image captured at a reference time; performing a second color correction with higher performance than the first color correction when state information including at least one of the position and orientation of the camera changes by a predetermined value or more from the state information at the reference time; updating the first time at which the second color correction was performed to a new reference time; outputting the color-corrected image by the second color correction at the reference time; and outputting the color-corrected image by the first color correction at times other than the reference time.

2. The color correction method according to claim 1, wherein the second color correction is performed by inputting an image taken by the imaging device before color correction into a machine learning model trained to perform color correction on an input image, thereby obtaining a color-corrected image.

3. The color correction method according to claim 1, wherein the first color correction is performed by converting the image before color correction to the image after color correction using a correction coefficient calculated from the image before color correction and the image after color correction in the second color correction.

4. The color correction method according to claim 3, wherein the calculation of the correction coefficient is started a predetermined time before a second time in which the state information changes by a predetermined value or more, based on the state information at multiple time points after the reference time.

5. The color correction method according to claim 3, wherein, when performing color correction after acquiring a predetermined amount of video footage captured by the imaging device, the correction coefficient for the first color correction performed between the third time and the fourth time for performing the second color correction is an interpolated value between the correction coefficient calculated at the third time and the correction coefficient calculated at the fourth time.

6. The color correction method according to claim 5, wherein the shooting environment conditions at the third time and the correction coefficient calculated at the third time are input into a prediction model to calculate the correction coefficient for the first color correction performed between the third time and the fourth time.

7. The color correction method according to claim 6, wherein the prediction model is a state transition model, and a state vector including a quaternion representing a direction vector and an angle using the correction coefficient is input to the prediction model.

8. A color correction method according to any one of claims 1 to 3, wherein a change in the state information of the imaging device is determined based on data detected by the imaging device or a sensor attached to a device on which the imaging device is mounted.

9. A color correction method according to any one of claims 1 to 3, wherein a change in the state information of the imaging device is determined based on a change in the distribution of each RGB component in an image captured by the imaging device.

10. The color correction method according to any one of claims 1 to 3, wherein, if the imaging device is capable of measuring depth, a change in the state information of the imaging device is determined based on a change in the distribution of depth information in an image captured by the imaging device.

11. The color correction method according to claim 9, wherein the change in the distribution is determined based on the change in the distribution near the center of the image.

12. A color correction program for causing a computer to perform a process that includes: performing a first color correction on each frame of video captured by a shooting device based on an image captured at a reference time; performing a second color correction with higher performance than the first color correction when state information including at least one of the position and orientation of the shooting device changes by a predetermined value or more from the state information at the reference time; updating the first time at which the second color correction was performed to a new reference time; outputting the color-corrected image by the second color correction at the reference time; and outputting the color-corrected image by the first color correction at times other than the reference time.

13. The color correction program according to claim 12, wherein the second color correction is performed by inputting an image taken by the imaging device before color correction into a machine learning model trained to perform color correction on an input image, thereby obtaining a color-corrected image.

14. The color correction program according to claim 12, wherein the first color correction includes converting the image before color correction to the image after color correction using a correction coefficient calculated from the image before color correction and the image after color correction in the second color correction.

15. The color correction program according to claim 14, wherein the calculation of the correction coefficient is started a predetermined time before a second time in which the state information changes by a predetermined value or more, based on the state information at multiple time points after the reference time.

16. The color correction program according to claim 14, wherein, when performing color correction after acquiring a predetermined amount of video footage captured by the imaging device, the correction coefficient for the first color correction performed between the third time and the fourth time for performing the second color correction is an interpolated value between the correction coefficient calculated at the third time and the correction coefficient calculated at the fourth time.

17. A color correction program according to claim 16, which inputs the shooting environment conditions at the third time and the correction coefficient calculated at the third time into a prediction model to calculate the correction coefficient for the first color correction performed between the third time and the fourth time.

18. The color correction program according to claim 17, wherein the prediction model is a state transition model, and a state vector including a quaternion representing a direction vector and an angle using the correction coefficient is input to the prediction model.

19. A color correction program according to any one of claims 12 to 14, which determines a change in the state information of the imaging device based on data detected by the imaging device or a device on which the imaging device is mounted.

20. A color correction program according to any one of claims 12 to 14, wherein a change in the state information of the imaging device is determined based on a change in the distribution of each RGB component in an image captured by the imaging device.

Images