Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises a processing parameter estimation means, a processing means, and a repetition control means. The processing parameter estimation means estimates a processing parameter on the basis of a state of an image of interest. By using the processing parameter, the processing means performs image processing on the image of interest to generate a processed image. The repetition control means sets the generated processed image as an image of interest and controls repetition of acquisition of a state of the image of interest, estimation of a processing parameter, and generation of a processed image.

Description

Information processing device, information processing method, and recording medium

[0001] This invention relates to an information processing device, an information processing method, and a recording medium.

[0002] For example, Patent Document 1 discloses a technique for improving image quality. The image processing apparatus described in Patent Document 1 includes an inference unit and a correction unit. When an input image is input, the inference unit infers the result of image processing to improve the image quality of the input image, and outputs the resulting image or feature quantity and a certainty based on the degree of uncertainty of the inference of the image processing result. The correction unit corrects the resulting image or feature quantity based on the certainty.

[0003] For example, according to Patent Document 1, the inference unit uses a trained model that uses machine learning to infer a denoised image and its certainty, and outputs the image processing result and the certainty as the degree of certainty of the inference for that image processing result. The certainty described in Patent Document 1 is a score that is small when the variance or error value of the model inference result is large, and large when the variance or error value of the model inference result is small.

[0004] For example, according to Patent Document 1, the correction unit corrects the denoised image, which is the result of image processing output by the inference unit, by performing a convolution operation using a kernel whose weights are the certainty map output by the inference unit.

[0005] Japanese Patent Publication No. 2024-98754

[0006] However, the state of the input image can change in various ways depending on the shooting environment, etc. For example, if the input image is taken with a camera installed outdoors, even if the images are taken with the same camera, the state of the input image can change significantly throughout the day due to the movement of the sun, changes in weather, etc. In the technology described in Patent Document 1, even if the denoised image is corrected by a convolution operation using a kernel with a certainty map as weights, it may be difficult to sufficiently improve the image quality.

[0007] One of the challenges of this disclosure is to improve image quality.

[0008] An information processing apparatus according to an aspect of the present disclosure includes: a processing parameter estimation unit that estimates a processing parameter based on a state of a target image; a processing unit that executes image processing on the target image using the processing parameter to generate a processed image; and an iterative control unit that controls repetition of acquisition of the state of the target image, estimation of the processing parameter, and generation of the processed image with the generated processed image as the target image.

[0009] An information processing method according to an aspect of the present disclosure includes: one or more computers estimating a processing parameter based on a state of a target image, executing image processing on the target image using the processing parameter to generate a processed image, and repeating acquisition of the state of the target image, estimation of the processing parameter, and generation of the processed image with the generated processed image as the target image to control the repetition.

[0010] A recording medium according to an aspect of the present disclosure has recorded thereon a program for causing one or more computers to estimate a processing parameter based on a state of a target image, execute image processing on the target image using the processing parameter to generate a processed image, and repeat acquisition of the state of the target image, estimation of the processing parameter, and generation of the processed image with the generated processed image as the target image to control the repetition.

[0011] According to an example of the present disclosure, it becomes possible to improve the state of an image.

[0012] This is a block diagram showing an example configuration of the first information processing device according to this disclosure. This is a flowchart showing an example of the processing operation of the first information processing device according to this disclosure. This is a block diagram showing a detailed example configuration of the first information processing device according to this disclosure. This is a flowchart showing a detailed example of the processing operation of the first information processing device according to this disclosure. This is a block diagram showing an example of the physical configuration of the first information processing device according to this disclosure. This is a block diagram showing an example configuration of the second information processing device according to this disclosure. This is a flowchart showing an example of the processing operation of the second information processing device according to this disclosure. This is a diagram showing an example of iterative processing of the state of a target image, calculation of a state evaluation value, image processing according to the state of the target image, and calculation of an image evaluation value according to this disclosure. This is a diagram showing an example of a target image and text associated with the target image. This is a block diagram showing an example configuration of the third information processing device according to this disclosure. This is a flowchart showing an example of the processing operation of the third information processing device according to this disclosure.

[0013] In this disclosure, the drawings are associated with one or more embodiments. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted where appropriate.

[0014] <Embodiment 1> As shown in Figure 1, the information processing device 100 according to the present disclosure comprises a processing parameter estimation unit 110, an image processing unit 120, and a repeat control unit 130.

[0015] The processing parameter estimation unit 110 estimates processing parameters based on the state of the target image. The image processing unit 120 performs image processing on the target image using the processing parameters to generate a processed image. The repeat control unit 130 uses the generated processed image as the target image and controls the repetition of acquiring the state of the target image, estimating processing parameters, and generating the processed image.

[0016] The information processing device 100 performs the information processing shown in Figure 2.

[0017] The processing parameter estimation unit 110 estimates processing parameters based on the state of the target image (step S110). The image processing unit 120 performs image processing on the target image using the processing parameters to generate a processed image (step S120). The repeat control unit 130 controls the repetition of steps S110 and S120 using the generated processed image as the target image (loop A: step S130). For example, the repeat control unit 130 repeatedly executes steps S110 and S120 until a predetermined termination condition is met.

[0018] This information processing device 100 allows for repeated image processing of a target image using processing parameters appropriate to its state, while simultaneously checking the state of the target image. Therefore, it becomes possible to improve the image's state.

[0019] The following describes a detailed example of the information processing device 100.

[0020] (Detailed example) The information processing device 100 generates an output image by repeatedly performing the estimation of processing parameters based on the state of the input image and the image processing using the estimated processing parameters. Then, the information processing device 100 performs image utilization processing using the output image.

[0021] Image utilization processing includes, for example, object detection processing that detects objects contained in an image, and biometric authentication processing that uses faces, irises, etc., contained in an image, at least a part of these processes. An object includes at least one of a person or an object. Note that image utilization processing is not limited to object detection processing or biometric authentication processing.

[0022] As shown in Figure 3, for example, the information processing device 100 includes, in addition to the processing parameter estimation unit 110, image processing unit 120, and repetition control unit 130 described above, an input image acquisition unit 105 and an image utilization unit 140.

[0023] The input image acquisition unit 105 acquires the input image. The image utilization unit 140 performs image utilization processing using the output image.

[0024] The information processing device 100 performs information processing as shown in Figure 4, for example. In Figure 4, steps S110a and S110b, steps S120a and S120b, and step S130a are detailed examples of steps S110, S120, and S130 described above, respectively.

[0025] This information processing is initiated, for example, when the input image acquisition unit 105 acquires an input image. However, the trigger for initiating information processing is not limited to this.

[0026] The processing parameter estimation unit 110 uses the input image as the target image and estimates the processing parameters based on the state of the target image (step S110a).

[0027] The image processing unit 120 uses the input image as the target image and performs image processing on the target image using the processing parameters estimated in step S110a to generate a processed image (step S120a).

[0028] The repeat control unit 130 controls the repetition of steps S110b and S120b using the generated processed image as the target image (loop A: step S130a).

[0029] The processing parameter estimation unit 110 uses the generated processed image as the target image and estimates the processing parameters based on the state of the target image (step S110b).

[0030] The image processing unit 120 uses the generated processed image as the target image and performs image processing on the target image using the processing parameters estimated in step S110b to generate a processed image (step S120b).

[0031] The generated processed image may be the processed image generated in step S120a in the first execution of loop A (step S130a). In subsequent executions of loop A (step S130a), the generated processed image may be the processed image generated in the previous step S120b.

[0032] The loop control unit 130 terminates loop A (step S130a) when, for example, a predetermined termination condition is met. The image utilization unit 140 performs image utilization processing using the output image (step S140).

[0033] From here, we will explain in more detail the functions of each functional unit and the processes that each functional unit performs.

[0034] (Regarding the input image acquisition unit 105) The input image acquisition unit 105 acquires the input image.

[0035] The input image acquisition unit 105 acquires input images from, for example, a shooting device (not shown), an image storage unit, etc. The shooting device may be a camera installed along a road, on a building, etc., an in-vehicle camera, a camera that takes images for biometric authentication provided on various devices, etc. The image storage unit is a storage unit that stores images in advance.

[0036] The input image is not limited to a still image; it may also be a video. In this case, the input image acquisition unit 105 acquires multiple frame images that make up the video.

[0037] (Regarding the processing parameter estimation unit 110) The processing parameter estimation unit 110 acquires the state of the target image based on the target image. Then, based on the state of the target image, the processing parameter estimation unit 110 estimates the processing parameters to be used for image processing of the target image.

[0038] (Regarding the state of the target image) The state of the target image may include, for example, the target image itself, a modified image obtained by transforming the target image, the difference between the target image and the previous target image, and a value obtained by combining the differences between the target image and the previous target image.

[0039] A deformed image is an image obtained by deforming the target image. Image deformation here may include, for example, downsampling the size of the target image to a predetermined size, or cutting out a region of a predetermined size from the target image, or at least one of these.

[0040] The difference between the target image and the previous target image is, for example, a tensor composed of the differences in pixel values of corresponding pixels between the target image and the previous target image. Specifically, for example, the difference v ｔ in the t-th series of processes ｔ is represented by x ｔ－１ − x

[0041] Here, the series of processes includes, for example, steps S110 and S120 described above, and specifically includes, for example, acquisition of the state of the target image, estimation of processing parameters, and image processing using the processing parameters, and the same applies hereinafter.

[0042] x ｔ represents a tensor composed of the pixel values of the target image in the t-th series of processes. x ｔ－１ represents a tensor composed of the pixel values of the target image in the (t - 1)-th series of processes. x ０ is, for example, a tensor with the value of all components being 0.

[0043] A tensor is an n-dimensional array (n is an integer greater than or equal to 0), and the same applies hereinafter. When n is 0, 1, or 2, the tensors are generally referred to as scalars, vectors, matrices, etc., respectively.

[0044] The value obtained by integrating the difference between the target image and the previous target image is, for example, a tensor obtained by integrating the target image and the difference between the target image and the previous target image in the channel direction. For example, assume that the target image is a two-dimensional image with height h and width w, and the channels are three components of red (R), green (G), and blue (B). In this case, each of x ｔ and v ｔ is a tensor composed of components of height h × width w × channel 3. The value obtained by integrating x ｔ and v ｔ is, for example, a tensor of height h × width w × channel 6 composed of the component corresponding to x ｔ (height h × width w × channel 3) and the component corresponding to v ｔ (height h × width w × channel 3).

[0045] (Regarding processing parameters) Processing parameters are parameters used by the image processing unit 120 for image processing of the target image. Processing parameters may include at least one of the following: parameters for determining the tone curve shape of the target image, parameters for deforming the histogram of the target image, parameters for reducing noise in the target image, and parameters related to the convolution kernel for performing convolution processing on the target image.

[0046] The processing parameter estimation unit 110 may estimate the processing parameters using, for example, a trained processing parameter estimation model. The processing parameter estimation model is a machine learning model for estimating processing parameters, and for example, when the state of a target image is input, it outputs the processing parameters of the target image.

[0047] For example, the processing parameter estimation unit 110 may include a trained processing parameter estimation model. When the processing parameter estimation unit 110 acquires a state based on the target image, it may input the state of the target image into the processing parameter estimation model to estimate the processing parameters of the target image.

[0048] The processing parameter estimation model may be stored in another information processing device (not shown), and the processing parameter estimation unit 110 may use this processing parameter estimation model to estimate the processing parameters. The training method for the processing parameter estimation model will be described in other embodiments.

[0049] (Regarding the image processing unit 120) The image processing unit 120 uses the estimated processing parameters for the target image to perform image processing on the target image and generate a processed image.

[0050] Image processing includes processing according to processing parameters. More specifically, the image processing performed by the image processing unit 120 on the target image includes processing according to the processing parameters estimated by the processing parameter estimation unit 110. In other words, the types of parameters included in the processing parameters estimated by the processing parameter estimation unit 110 are determined according to the image processing performed by the image processing unit 120 on the target image.

[0051] For example, if the processing parameters include a parameter that determines the tone curve shape of the target image, the image processing includes a process to change the tone curve shape of the target image. In this case, the image processing unit 120 processes the target image using the parameter that determines the tone curve shape of the target image. This generates a processed image with a tone curve shape corresponding to the parameter. The tone curve shape may be defined, for example, by a Bézier curve.

[0052] For example, if the processing parameters include parameters that deform the histogram of the target image, the image processing includes processing that deforms the histogram of the target image. In this case, the image processing unit 120 processes the target image using the parameters that deform the histogram of the target image. This generates a processed image of the histogram corresponding to those parameters.

[0053] For example, if the processing parameters include parameters for reducing noise in the target image, the image processing includes processing to reduce noise in the target image. In this case, the image processing unit 120 processes the target image using the parameters for reducing noise in the target image. This generates a processed image with reduced noise according to those parameters.

[0054] For example, if the processing parameters include parameters related to a convolution kernel for performing convolution on the target image, the image processing includes performing convolution on the target image. In this case, the image processing unit 120 processes the target image using the parameters related to the convolution kernel. This generates a processed image by performing convolution using the parameters.

[0055] (Regarding the repetition control unit 130) The repetition control unit 130 causes the processing parameter estimation unit 110 and the image processing unit 120 to repeatedly execute a series of processes on the target image until, for example, a predetermined termination condition is met. In other words, the processing parameter estimation unit 110 and the image processing unit 120 repeatedly execute a series of processes under the control of the repetition control unit 130.

[0056] In the first series of processes executed, i.e., in steps S110a and S120a, the target image is, for example, the input image. In subsequent series of processes executed, i.e., in steps S110b and S120b, the target image is, for example, the processed image generated in the previous series of processes. The processed image generated in the previous series of processes is, for example, the processed image generated in step S120a, or the processed image generated in the previous step S120b.

[0057] The termination condition may be, for example, the execution of a series of processes a predetermined number of times. In this case, the number of repetitions of the series of processes, including the series of processes using the input image as the target image, is at least two. That is, steps S110a and S120a are executed, followed by steps S110b and S120b being executed at least once. Note that the termination condition is not limited to this, and other examples will be described later.

[0058] When the termination condition is met, the loop control unit 130 causes, for example, the image processing unit 120 to output an output image. The loop control unit 130 may also output the output image itself. The output image is the processed image generated in the last executed series of processes, that is, for example, the processed image generated in the last executed step S120b.

[0059] (Regarding the image utilization unit 140) As described above, the image utilization unit 140 performs image utilization processing using the output image. The output image is, for example, the last processed image generated. The image utilization unit 140 performs image utilization processing using, for example, the output image and the image utilization model, which is a trained machine learning model.

[0060] For example, if the image utilization process is an object detection process that detects objects contained in an image, the image utilization model may be an object detection model that detects objects contained in an image. The object detection model takes an image as input, detects objects contained in the image, and outputs a detection result. The detection result includes at least one of the following: for example, the position of the object in the image, the type of object, etc. The object detection model may be constructed using general techniques.

[0061] For example, if the image processing is biometric authentication, the image processing model may be a feature extraction model that extracts feature quantities from predetermined parts of an image, such as a face or iris. The feature extraction model takes an image as input and extracts feature quantities from predetermined parts of the image. In the image processing, for example, the extracted feature quantities of the predetermined parts are used to authenticate whether or not those parts belong to a person or other entity that has been registered in advance. The feature extraction model may be constructed using general technologies.

[0062] The object detection model and feature extraction model are examples of image utilization models used when the image utilization process is object detection or biometric authentication, respectively, and the image utilization model is not limited to these. The image utilization model may be, for example, any appropriate machine learning model depending on the image utilization process.

[0063] (Example of physical configuration of the information processing device 100) Figure 5 shows an example of the physical configuration of the information processing device 100. Physically, the information processing device 100 includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0064] Bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070 to send and receive data to and from each other. However, the method of connecting the processor 1020 and the other components to each other is not limited to bus connection.

[0065] Processor 1020 is a processor implemented using components such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0066] Memory 1030 is a main memory device implemented as RAM (Random Access Memory), etc.

[0067] The storage device 1040 is an auxiliary storage device implemented as an HDD (Hard Disk Drive), SSD (Solid State Drive), memory card, or ROM (Read Only Memory). The storage device 1040 stores program modules for realizing the functions of the device equipped with it. The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing the functions corresponding to those program modules.

[0068] The network interface 1050 is an interface for connecting a device equipped with it to a communication network.

[0069] The input interface 1060 is an interface for the user to input information. The input interface 1060 consists of, for example, a touch panel, a keyboard, a mouse, and the like.

[0070] The output interface 1070 is an interface for presenting information to the user. The output interface 1070 is composed of, for example, a liquid crystal panel, an organic EL (Electro-Luminescence) panel, and the like.

[0071] Embodiment 1 has been described above.

[0072] (Function and Effects) According to this embodiment, the information processing device 100 comprises a processing parameter estimation unit 110, an image processing unit 120, and a repeat control unit 130.

[0073] The processing parameter estimation unit 110 estimates processing parameters based on the state of the target image. The image processing unit 120 performs image processing on the target image using the processing parameters to generate a processed image. The repeat control unit 130 uses the generated processed image as the target image and controls the repetition of acquiring the state of the target image, estimating processing parameters, and generating the processed image.

[0074] This allows for repeated image processing of the target image using processing parameters appropriate to its state, while monitoring its condition. Consequently, it becomes possible to improve the image's quality.

[0075] According to this embodiment, the state of the target image includes at least one of the following: the target image, a deformed image obtained by deforming the target image, the difference between the current target image and the previous target image, and a value obtained by integrating the differences between the current target image and the previous target image.

[0076] This allows for repeated image processing of the target image using processing parameters appropriate to its state, while monitoring its condition. Consequently, it becomes possible to improve the image's quality.

[0077] According to this embodiment, the processing parameters include at least one of the following: a parameter for determining the tone curve shape of the target image; a parameter for deforming the histogram of the target image; a parameter for reducing noise in the target image; and a parameter for a convolution kernel for performing convolution processing on the target image. The image processing includes processing according to the processing parameters.

[0078] This allows for the repeated application of at least one of the following processes to the target image: changing the tone curve shape, deforming the histogram of the target image, reducing noise in the target image, or performing a convolution on the target image. Consequently, it becomes possible to improve the image quality.

[0079] <Embodiment 2> Embodiment 2 describes an example of a training method for a processing parameter estimation model. In this embodiment, an example of using reinforcement learning to train the processing parameter estimation model is described. The training of the processing parameter estimation model is not limited to that described in this embodiment, and reinforcement learning methods such as Soft Actor Critic (SAC) and Deep Deterministic Policy Gradient (DDPG) may be used.

[0080] In this embodiment, in order to simplify the explanation, descriptions that overlap with other embodiments will be omitted as appropriate.

[0081] As shown in Figure 6, for example, the information processing device 200 includes a training image acquisition unit 205, a processing parameter estimation unit 210, an image processing unit 220, an image evaluation unit 250, a state evaluation estimation unit 260, an update unit 270, and a training control unit 280.

[0082] The training image acquisition unit 205 acquires training images, which are images used for training.

[0083] The processing parameter estimation unit 210 estimates processing parameters based on the state of the target image.

[0084] The image processing unit 220 performs image processing on the target image using processing parameters to generate a processed image.

[0085] The image evaluation unit 250 uses an evaluation function for evaluating images to obtain an image evaluation value for the target image.

[0086] The state evaluation estimation unit 260 estimates a state evaluation value, which is the expected value of the sum of image evaluation values, based on the state of the target image and the processing parameters estimated for the target image.

[0087] The update unit 270 updates the first parameter used for estimating processing parameters and the second parameter used for estimating state evaluation values, using the image evaluation value and the state evaluation value.

[0088] The training control unit 280 controls the repetition of a series of training processes using the generated processed image as the target image. For example, the training control unit 280 repeatedly executes the series of training processes until a predetermined training termination condition is met.

[0089] A series of training processes is a series of processes for training, and the same applies hereafter. A series of training processes includes, for example, obtaining the state of a target image, estimating processing parameters, generating a processed image, obtaining image evaluation values, estimating state evaluation values, and updating the first and second parameters.

[0090] The information processing device 200 performs information processing as shown in Figure 7, for example. This information processing is started, for example, when the training image acquisition unit 205 acquires a training image. However, the trigger for starting the information processing is not limited to this.

[0091] The processing parameter estimation unit 210 uses the training image as the target image and estimates the processing parameters based on the state of the target image (step S210a).

[0092] The image processing unit 220 uses the training image as the target image and performs image processing on the target image using processing parameters to generate a processed image (step S220a).

[0093] The training control unit 280 controls the repetition of steps S210b, S220b, S250, S260, and S270 using the generated processed image as the target image (loop B: step S280).

[0094] The processing parameter estimation unit 210 uses the generated processed image as the target image and estimates the processing parameters based on the state of the target image (step S210b).

[0095] The image processing unit 220 uses the generated processed image as the target image and performs image processing on the target image using processing parameters to generate a processed image (step S220b).

[0096] Here, where t is a non-negative integer, the "generated processed image" in steps S210b and S220b executed in the t+1th loop B (step S280) may be the processed image generated in the previous image processing. That is, the "generated processed image" in steps S210b and S220b executed in the t+1th loop B (step S280) may be the processed image generated in the tth image processing.

[0097] Note that the 0th image processing step is the image processing performed in step S220a, and the same applies hereafter.

[0098] The image evaluation unit 250 uses an evaluation function for evaluating images to obtain an image evaluation value for the generated processed image as the target image (step S250).

[0099] Here, the "generated processed image" in step S250, which is executed in the t+1th loop B (step S280), may be the processed image generated in the current image processing. That is, the "generated processed image" in step S250, which is executed in the t+1th loop B (step S280), may be the processed image generated in the t+1th image processing.

[0100] The state evaluation estimation unit 260 uses the generated processed image as the target image and estimates a state evaluation value, which is the expected value of the sum of image evaluation values, based on the state of the target image and the processing parameters estimated for the target image (step S260).

[0101] Here, the "generated processed image" in step S260, which is executed in the t+1th loop B (step S280), may be the processed image generated in the previous and current image processing. That is, the state evaluation estimation unit 260 may estimate a state evaluation value based on the state of each processed image generated in the tth and t+1th image processing, and the processing parameters estimated for each processed image generated in the tth and t+1th image processing.

[0102] The update unit 270 updates the first parameter used for estimating the processing parameter and the second parameter used for estimating the state evaluation value using the image evaluation value acquired in step S250 and the state evaluation value estimated in S260 (step S270).

[0103] Here, the image evaluation value and state evaluation value used in step S270, which is executed in the t+1th loop B (step S280), may be the image evaluation value and estimated state evaluation value obtained in the current loop B, respectively.

[0104] In other words, the image evaluation value used in step S270, which is executed in the t+1th loop B (step S280), may be the image evaluation value obtained in step S250, which was executed in the t+1th time. The state evaluation value used in step S270, which is executed in the t+1th loop B (step S280), may be the state evaluation value estimated in step S260, which was executed in the t+1th time.

[0105] The first parameter is, for example, a parameter included in a processing parameter estimation model that estimates processing parameters. The second parameter is, for example, a parameter included in a state evaluation model that estimates state evaluation values.

[0106] When the training control unit 280 meets the predetermined training termination conditions, it terminates loop B (step S280) and ends the information processing using the training image. There may be multiple training images, in which case the information processing is repeatedly performed using each of the training images.

[0107] By performing this information processing, the state s of the target image becomes as shown in Figure 8. ｔ Image evaluation value r ｔ Calculation of state s ｔ Image processing a according to the method ｔ = π(s ｔ ), state evaluation value Q π (s ｔ , a ｔ The calculation of the state evaluation value Q is repeated. π (s ｔ , a ｔ As mentioned above, Q is the expected value of the sum of the image evaluation values, and can also be said to be the expected value of the return obtained by the subsequent policy π when action a is performed starting from state s. Thus, the state evaluation value Q π (s ｔ , a ｔ ), that is, the processing parameter estimation model and the state evaluation model can be trained simultaneously to maximize the expected value of the sum of the image evaluation values.

[0108] The following describes detailed examples of the functions and processes of the information processing device 200.

[0109] (Regarding the training image acquisition unit 205) The training image acquisition unit 205 acquires training images as described above. The training images are images for training and are prepared in advance. For example, the training images are stored in advance in a storage unit (not shown). The training image acquisition unit 205 acquires the training images from the storage unit. If there are multiple training images, the training image acquisition unit 205 may acquire the training images one by one. In this way, the information processing illustrated in Figure 7 may be repeatedly executed using each of the training images.

[0110] (Regarding the processing parameter estimation unit 210) The processing parameter estimation unit 210 acquires the state of the target image based on the target image. Then, based on the state of the target image, the processing parameter estimation unit 210 estimates the processing parameters to be used for image processing of the target image.

[0111] For example, while the processing parameter estimation unit 110 uses a trained processing parameter estimation model, the processing parameter estimation unit 210 uses an untrained processing parameter estimation model to estimate the processing parameters. Except for this point, the processing parameter estimation unit 210 may be the same as the processing parameter estimation unit 110 described above.

[0112] For example, the processing parameter estimation model is based on the state s of the target image generated in the tth image processing step. ｔ When you input the processing parameter a of the target image, ｔ The output is as follows: The processing parameter estimation model may be constructed, for example, by a convolutional neural network.

[0113] The processing parameter estimation model includes a first parameter, which is used to estimate the processing parameters. The processing parameter estimation model is trained by updating the first parameter. As a result, a trained processing parameter estimation model is constructed.

[0114] (Regarding the image processing unit 220) The image processing unit 220, similar to the image processing unit 120 described above, uses the estimated processing parameters for the target image to perform image processing on the target image and generate a processed image. The image processing performed by the image processing unit 220 may be the same as the image processing performed by the image processing unit 120 described above.

[0115] (Regarding the image evaluation unit 250) As described above, the image evaluation unit 250 obtains an image evaluation value by evaluating the target image using an evaluation function. The evaluation function f is a function for evaluating an image and is predetermined, for example. The evaluation function takes the state of the target image as input and outputs an image evaluation value for the target image.

[0116] Image evaluation value r of the target image ｔ This may include, for example, at least one of the following (1) to (5).

[0117] (1) Statistical value r of the image characteristic value that indicates the image characteristics of the target image ｔ (2) Loss r based on the loss function used to train the image utilization model that uses the processed image ｔ (3) Similarity r of feature quantities obtained using a multimodal model from each of the target image and the word associated with the target image. ｔ (4) Loss r obtained from the probability distribution of the similarity of features obtained using a multimodal model from each of the target image and the text associated with the target image. ｔ (5) state s ｔ The image evaluation function g(s) gives a larger value for better images. ｔ When using ), the state of the target image this time and last time s ｔ The difference r is obtained by inputting the image evaluation function g into the image evaluation function. ｔ (= g(s ｔ )-g(s ｔ－１ ))

[0118] For example, in the case of the image evaluation value described in (1) above, the evaluation function may include a function that, when a target image is input, outputs statistical values ​​of image characteristic values ​​that indicate the image characteristics of the target image. The image characteristic values ​​of the target image may include one or more of the following, for example, the contrast value and the average brightness value of the target image.

[0119] For example, in the case of the image evaluation value described in (2) above, the evaluation function may be the loss function used to train the image utilization model that uses the processed image. This evaluation function takes a target image as input and outputs a loss corresponding to that target image.

[0120] For example, in the case of the image evaluation value described in (3) above, the evaluation function may include a function that outputs the similarity of features obtained using a multimodal model from both the target image and the words associated with the target image.

[0121] An example of a multimodal model is CLIP (Contrastive Language-Image Pretraining). A multimodal model outputs, for example, features extracted from an image and features of a word associated with that image. In a multimodal model, the more similar the content of the image and the word are, the more similar their respective features become. Therefore, the similarity of the features obtained from the target image and the word associated with that image using a multimodal model will be a larger value as the target image and the word become more similar. The similarity can be, for example, cosine similarity.

[0122] For example, in the case of the image evaluation value described in (4) above, the evaluation function may be a function that outputs a loss obtained from the probability distribution of the similarity of features obtained using a multimodal model from both the target image and the text associated with the target image.

[0123] A sentence is composed of multiple words. Similarity can be measured using cosine similarity, for example. For instance, suppose a target image containing a dog, as shown in Figure 9, is associated with the sentence, "A bright image of a large dog." From the image features and sentence features extracted from the target image and sentence respectively, a probability distribution of the similarity between the image features and sentence features can be obtained, and the loss can be calculated.

[0124] For example, in the case of the image evaluation value in (5) above, the evaluation function is the state s of the target image this time and last time. ｔ The difference r is obtained by inputting the image evaluation function g into the image evaluation function. ｔ (= g(s ｔ)-g(s ｔ－１ The image evaluation function g(s) may include a function that outputs ). ｔ ) is state s ｔ This function gives a larger value for better images.

[0125] (Regarding the state evaluation estimation unit 260) As described above, the state evaluation estimation unit 260 estimates a state evaluation value, which is the expected value of the sum of image evaluation values, based on the state of the target image and the processing parameters estimated for the target image.

[0126] For example, the state evaluation estimation unit 260 estimates the state evaluation value using a state evaluation function. The state evaluation function is a machine learning model that estimates the state evaluation value.

[0127] For example, the state evaluation model is the state s of the target image generated in the tth image processing step. ｔ And the state s ｔ Image processing using processing parameters estimated based on a ｔ When you input this, the state evaluation value q ｔ The output is as follows: The state evaluation model may be constructed, for example, by a convolutional neural network.

[0128] The state evaluation model includes a second parameter, which is used to estimate the state evaluation value. The state evaluation model is trained by updating the second parameter, resulting in the construction of a trained state evaluation model.

[0129] (Regarding the update unit 270) As described above, the update unit 270 updates the first parameter used for estimating the processing parameters and the second parameter used for estimating the state evaluation values, using the image evaluation values ​​and the state evaluation values.

[0130] For example, the update unit 270 has a first loss L ｑ and the second loss L π Calculate the result.

[0131] The first loss Lq is the loss used to update the first parameter and is represented, for example, by equation (1) below. The second loss Lπ is the loss used to update the first parameter and is represented, for example, by equation (2) below. γ in equation (1) and α in equation (2) are hyperparameters. Note that the first loss Lq and the second loss Lπ are not limited to those represented by equations (1) and (2) below.

[0132] First loss Lq=(Q π (s ｔ , a ｔ ) - (r ｔ +γQ π (s ｔ＋１ , a ｔ＋１ )) ２ ...Formula (1)

[0133] Second loss Lπ=αlog(π(s ｔ ) - Q π (s ｔ , a ｔ ) ...Formula (2)

[0134] The update unit 270 then updates the first and second parameters so that the first loss Lq and the second loss Lπ are reduced, respectively. A gradient method such as Adam (Adaptive Moment Estimation) or SGD (stochastic gradient descend) may be used to calculate the first and second parameters for reducing the first loss Lq and the second loss Lπ, respectively.

[0135] (Regarding the training control unit 280) The training control unit 280 repeatedly causes the processing parameter estimation unit 210, image processing unit 220, image evaluation unit 250, state evaluation estimation unit 260, and update unit 270 to execute a series of training processes on the target image, for example, until predetermined training termination conditions are met. In other words, the processing parameter estimation unit 210, image processing unit 220, image evaluation unit 250, state evaluation estimation unit 260, and update unit 270 repeatedly execute a series of training processes under the control of the training control unit 280.

[0136] The series of training processes includes, for example, the steps S210a, S220a, S210b, S220b, S250, S260, and S270 described above. More specifically, the series of training processes includes, for example, acquiring the state of the target image, estimating processing parameters, image processing using the processing parameters (generating a processed image), acquiring image evaluation values, estimating state evaluation values, and updating the first and second parameters.

[0137] In the initial series of training processes, specifically in steps S210a and S220a, the target image is, for example, a training image.

[0138] In the subsequent series of processes executed from the second time onward, i.e., in steps S210b, S220b, S250, S260, and S270, for example, the target image includes a processed image generated in the previous or current series of training processes. The processed image generated in the previous series of processes is, for example, the processed image generated in step S220a, or the processed image generated in the previous step S220b.

[0139] In detail, the target images in steps S210b and S220b, which are executed from the second time onward, are, for example, the processed images generated in the previous series of training processes. The target images in step S250, which is executed from the second time onward, are, for example, the processed images generated in the current series of training processes. The target images in step S260, which is executed from the second time onward, are, for example, the processed images generated in the previous and current series of training processes.

[0140] The training completion condition may be, for example, the execution of a series of training processes a predetermined number of times. In this case, the number of repetitions of the series of training processes, including the series of processes using the input image as the target image, is at least two. That is, after steps S210a and S220a are executed, steps S210b, S220b, S250, S260 and S270 are executed at least once.

[0141] The training termination condition may be, for example, that the image evaluation value obtained by evaluating the target image becomes equal to or greater than a predetermined threshold. The training termination condition may be, for example, the first loss L ｑ and the second loss L πIt is also possible that the value becomes below a predetermined threshold. In this case, the first loss L ｑ and the second loss L π Different thresholds may be set for each. The training termination condition is the first loss L. ｑ and the second loss L π It is also possible that both of these be below the threshold set for each, and the first loss L ｑ and the second loss L π At least one of these two values ​​may be below the threshold set for each.

[0142] Note that the training termination conditions are not limited to those exemplified here. Training termination conditions may include, for example, the image evaluation value of the target image exceeding a predetermined threshold, and the first loss L ｑ and the second loss L π This may include satisfying at least one of the following conditions: that the value is below a predetermined threshold.

[0143] The repeating control unit 130 terminates information processing using the training image when the termination condition is met. If there are training images that have not yet been used for training, information processing using those training images may be performed.

[0144] The information processing device 200 may be physically configured in the same way as the information processing device 100.

[0145] Embodiment 2 has been described above.

[0146] (Function and Effects) According to this embodiment, the information processing device 200 includes an image evaluation unit 250, a state evaluation estimation unit 260, and an update unit 270.

[0147] The image evaluation unit 250 obtains an image evaluation value by evaluating the target image using an evaluation function for evaluating the image. The state evaluation estimation unit 260 estimates a state evaluation value, which is the expected value of the sum of the image evaluation values, based on the state of the target image and the processing parameters estimated for the target image. The update unit 270 updates the first parameter used for estimating the processing parameters and the second parameter used for estimating the state evaluation value, using the image evaluation value and the state evaluation value.

[0148] This allows the first parameter to be updated, and a processing parameter estimation model can be trained to estimate processing parameters according to the state of the target image. Using the trained processing parameter estimation model, it is possible to estimate processing parameters for image processing according to the state of the target image while checking its state. Therefore, it becomes possible to improve the state of the image.

[0149] According to this embodiment, the image evaluation value of the target image includes at least one of the following: a statistical value of image characteristic values ​​indicating the image characteristics of the target image; a loss based on the loss function used to train the image utilization model that utilizes the processed image; the similarity of features obtained using a multimodal model from each of the target image and the words associated with the target image; a loss obtained from the probability distribution of the similarity of features obtained using a multimodal model from each of the target image and the sentences associated with the target image; and, in the case of using an image evaluation function in which images in better condition have larger values, the difference between the values ​​obtained by inputting the state of the target image this time and last time into the image evaluation function.

[0150] This allows the first parameter to be updated, and a processing parameter estimation model can be trained to estimate processing parameters according to the state of the target image. Using the trained processing parameter estimation model, it is possible to estimate processing parameters for image processing according to the state of the target image while checking its state. Therefore, it becomes possible to improve the state of the image.

[0151] <Embodiment 3> In Embodiment 3, as another example of the termination condition described in Embodiment 1, an example of a termination condition using an image evaluation value obtained by evaluating the target image will be described.

[0152] In this embodiment, in order to simplify the explanation, descriptions that overlap with other embodiments will be omitted as appropriate.

[0153] As shown in Figure 10, the information processing device 300 includes the processing parameter estimation unit 110 and image processing unit 120 described in Embodiment 1, the image evaluation unit 250 described in Embodiment 2, the repetition control unit 330, and the image utilization unit 140 described in Embodiment 1.

[0154] The repeat control unit 330, similar to the repeat control unit 130 described above, controls the repetition of a series of processes using the generated processed image as the target image until a predetermined termination condition is met. In this embodiment, the series of processes may include obtaining the state of the target image, estimating processing parameters, and image processing using the processing parameters, as well as obtaining an image evaluation value of the target image.

[0155] The termination conditions include, for example, the image evaluation value exceeding a predetermined threshold.

[0156] The termination condition may further include executing the series of processes a predetermined number of times. In this case, the repetition control unit 330 may repeatedly execute the series of processes until, for example, either the image evaluation value becomes equal to or above a predetermined threshold, or the image evaluation value becomes equal to or above a predetermined threshold.

[0157] The information processing device 300 performs the information processing shown in Figure 11.

[0158] Steps S110a and S120a are executed.

[0159] The repeat control unit 330 controls the repetition of steps S110b, S120b, and S250 using the generated processed image as the target image (loop C: step S330).

[0160] The generated processed image may be the processed image generated in step S120a in the first execution of loop C (step S330). In subsequent executions of loop C (step S330), the generated processed image may be the processed image generated in the previous step S120b.

[0161] The repeating control unit 330 terminates loop C (step S330) when, for example, a predetermined termination condition is met. Step S140 is then executed.

[0162] (Function and Effects) According to this embodiment, the information processing device 300 comprises an image evaluation unit 250 and a repeat control unit 330.

[0163] The image evaluation unit 250 uses an evaluation function to evaluate the image and obtains an image evaluation value for the target image. The repeat control unit 330 uses the generated processed image as the target image and repeatedly performs the following until the termination condition is met: obtaining the state of the target image, estimating processing parameters, generating the processed image, and obtaining the image evaluation value for the target image. The termination condition includes the image evaluation value being equal to or greater than a predetermined threshold.

[0164] This allows the system to monitor the state of the target image and repeatedly perform image processing on it using processing parameters appropriate to that state, until the image evaluation value exceeds a predetermined threshold. Consequently, it becomes possible to improve the image quality.

[0165] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0166] Furthermore, while the flowcharts used in the above description show multiple steps (processes) in sequence, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impair the content.

[0167] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0168] 1. An information processing apparatus comprising: processing parameter estimation means for estimating processing parameters based on the state of a target image; processing means for generating a processed image by performing image processing on the target image using the processing parameters; and repetition control means for controlling the repetition of acquiring the state of the target image, estimating the processing parameters, and generating the processed image, with the generated processed image as the target image. 2. The information processing apparatus according to 1, wherein the state of the target image includes at least one of the target image, a deformed image obtained by deforming the target image, the difference between the target image and the previous target image, and a value obtained by integrating the difference between the target image and the previous target image. 3. The information processing apparatus according to 1 or 2, wherein the processing parameters include at least one of a parameter for determining the tone curve shape of the target image, a parameter for deforming the histogram of the target image, a parameter for reducing noise in the target image, and a parameter relating to a convolution kernel for performing convolution processing on the target image, and the image processing includes processing according to the processing parameters. 4. An information processing apparatus according to 1. or 2., further comprising: an image evaluation means for acquiring an image evaluation value obtained by evaluating the target image using an evaluation function for evaluating the image, the repeat control means for repeatedly performing the acquisition of the state of the target image, estimation of processing parameters, generation of the processed image, and acquisition of the image evaluation value of the target image, using the generated processed image as the target image, until a termination condition is met, the termination condition includes the image evaluation value being equal to or greater than a predetermined threshold. 5. An information processing apparatus according to any one of 1. to 4., further comprising: an image evaluation means for acquiring an image evaluation value obtained by evaluating the target image using an evaluation function for evaluating the image; a state evaluation estimation means for estimating a state evaluation value which is the expected value of the sum of the image evaluation values ​​based on the state of the target image and the processing parameters estimated for the target image; and an update means for updating a first parameter used for estimating the processing parameters and a second parameter used for estimating the state evaluation value using the image evaluation value and the state evaluation value.6. The information processing apparatus according to 5. wherein the image evaluation value of the target image includes at least one of: a statistical value of image characteristic values ​​indicating the image characteristics of the target image; a loss based on a loss function used to train an image utilization model that utilizes the processed image; the similarity of feature quantities obtained using a multimodal model from each of the target image and the words associated with the target image; a loss obtained from the probability distribution of the similarity of feature quantities obtained using a multimodal model from each of the target image and the sentences associated with the target image; and the difference in values ​​obtained by inputting the current and previous states of the target image into the image evaluation function when an image evaluation function is used in which images with better states have larger values. 7. An information processing method in which one or more computers estimate processing parameters based on the state of a target image; perform image processing on the target image using the processing parameters to generate a processed image; and control the repetition of acquiring the state of the target image, estimating the processing parameters, and generating the processed image, with the generated processed image as the target image. 8. The information processing method according to 7. The state of the target image includes at least one of the target image, a deformed image obtained by deforming the target image, the difference between the target image and the previous target image, and a value obtained by integrating the difference between the target image and the previous target image. 9. The information processing method according to 7. or 8. The processing parameters include at least one of a parameter for determining the tone curve shape of the target image, a parameter for deforming the histogram of the target image, a parameter for reducing noise in the target image, and a parameter for a convolution kernel for performing convolution processing on the target image, and the image processing includes processing according to the processing parameters. 10. Furthermore, an image evaluation value is obtained by evaluating the target image using an evaluation function for evaluating the image, and in the iterative control, the generated processed image is used as the target image, and the acquisition of the state of the target image, estimation of processing parameters, generation of the processed image, and acquisition of the image evaluation value of the target image are repeatedly performed until a termination condition is met, the termination condition includes the image evaluation value being equal to or greater than a predetermined threshold.11. An information processing method according to any one of 7 to 10, further comprising: obtaining an image evaluation value obtained by evaluating the target image using an evaluation function for evaluating the image; estimating a state evaluation value which is the expected value of the sum of the image evaluation values ​​based on the state of the target image and the processing parameters estimated for the target image; and updating a first parameter used for estimating the processing parameters and a second parameter used for estimating the state evaluation value using the image evaluation value and the state evaluation value. 12. The image evaluation value of the target image includes at least one of: a statistical value of the image characteristic value indicating the image characteristics of the target image; a loss based on the loss function used to train an image utilization model that utilizes the processed image; the similarity of feature quantities obtained using a multimodal model from each of the target image and the words associated with the target image; a loss obtained from the probability distribution of the similarity of feature quantities obtained using a multimodal model from each of the target image and the sentences associated with the target image; and the difference between the values ​​obtained by inputting the current and previous states of the target image into the image evaluation function when an image evaluation function is used which has a larger value for images with better states. The information processing method described in 13. 13. A program that causes one or more computers to execute the information processing method described in any one of 7. to 12. 14. A recording medium on which the program described in 13. is recorded.

[0169] 100, 200, 300 Information processing device 105 Input image acquisition unit 110, 210 Processing parameter estimation unit 120, 220 Image processing unit 130, 330 Repeat control unit 140 Image utilization unit 205 Training image acquisition unit 250 Image evaluation unit 260 State evaluation estimation unit 270 Update unit 280 Training control unit

Claims

1. An information processing apparatus comprising: processing parameter estimation means for estimating processing parameters based on the state of a target image; processing means for generating a processed image by performing image processing on the target image using the processing parameters; and repetition control means for controlling the repetition of acquiring the state of the target image, estimating the processing parameters, and generating the processed image, with the generated processed image as the target image.

2. The information processing apparatus according to claim 1, wherein the state of the target image includes at least one of the target image, a deformed image obtained by deforming the target image, the difference between the target image and the previous target image, and a value obtained by integrating the difference between the target image and the previous target image.

3. The information processing apparatus according to claim 1 or 2, wherein the processing parameters include at least one of a parameter for determining the tone curve shape of the target image, a parameter for deforming the histogram of the target image, a parameter for reducing noise in the target image, and a parameter relating to a convolution kernel for performing convolution processing on the target image, and the image processing includes processing according to the processing parameters.

4. An information processing apparatus according to claim 1 or 2, further comprising an image evaluation means for evaluating a target image using an evaluation function for evaluating an image, wherein the iterative control means repeatedly performs the acquisition of the state of the target image, estimation of processing parameters, generation of a processed image, and acquisition of an image evaluation value of the target image, using the generated processed image as the target image, until a termination condition is met, the termination condition being that the image evaluation value is equal to or greater than a predetermined threshold.

5. An information processing apparatus according to any one of claims 1 to 4, further comprising: an image evaluation means for obtaining an image evaluation value obtained by evaluating a target image using an evaluation function for evaluating an image; a state evaluation estimation means for estimating a state evaluation value which is the expected value of the sum of the image evaluation values, based on the state of the target image and the processing parameters estimated for the target image; and an update means for updating a first parameter used for estimating the processing parameters and a second parameter used for estimating the state evaluation value, using the image evaluation value and the state evaluation value.

6. The information processing apparatus according to claim 5, wherein the image evaluation value of the target image includes at least one of: a statistical value of image characteristic values ​​indicating the image characteristics of the target image; a loss based on a loss function used to train an image utilization model that utilizes the processed image; the similarity of feature quantities obtained using a multimodal model from each of the target image and the words associated with the target image; a loss obtained from the probability distribution of the similarity of feature quantities obtained using a multimodal model from each of the target image and the sentences associated with the target image; and the difference between the values ​​obtained by inputting the current and previous states of the target image into the image evaluation function, when an image evaluation function is used in which images with better states have larger values.

7. An information processing method in which one or more computers estimate processing parameters based on the state of a target image, perform image processing on the target image using the processing parameters to generate a processed image, and control the repetition of acquiring the state of the target image, estimating the processing parameters, and generating the processed image, with the generated processed image as the target image.

8. A recording medium on which a program is stored that causes one or more computers to perform the following actions: estimate processing parameters based on the state of a target image; perform image processing on the target image using the processing parameters to generate a processed image; and control the repetition of acquiring the state of the target image, estimating the processing parameters, and generating the processed image, using the generated processed image as the target image.

Images