Information processing device, information processing method, and information processing program

Abstract

This information processing device comprises: an acquisition unit that acquires a first image that is the subject of a task; a correction processing unit that uses feature information of the first image to execute, in parallel, a plurality of correction processes, which are each different correction processes; and a generation unit that uses a plurality of second images, respectively obtained by the plurality of correction processes, to generate a third image which is to be used in a process related to the task.

Description

Information processing device, information processing method, and information processing program

[0001] This disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] For example, technologies have been provided for processing tasks such as object detection using images obtained by an image sensor, such as RAW (Raw Image Format) images (for example, Non-Patent Document 1).

[0003] Ruikang Xu, Chang Chen, Jingyang Peng, Cheng Li, Yibin Huang, Fenglong Song, Youliang Yan, Zhiwei Xiong, "Toward RAW Object Detection: A New Benchmark and A New Model", [online] [searched on November 1, 2024], Internet <URL: https: / / openaccess.thecvf.com / content / CVPR2023 / papers / Xu_Toward_RAW_Object_Detection_A_New_Benchmark_and_a_New_CVPR_2023_paper.pdf>

[0004] According to conventional technology, the dynamic range of an image is adjusted (corrected) in order to obtain high dynamic range data necessary for object detection algorithms to process various lighting conditions such as strong glare.

[0005] However, there is room for improvement in conventional technology. For example, while conventional technology can correct the dynamic range, it does not take into account the processing when multiple correction processes are performed, and it is not guaranteed that it can properly perform corrections when multiple correction processes are applied. Therefore, it is desirable to be able to properly generate images that reflect multiple correction processes.

[0006] Therefore, this disclosure proposes an information processing apparatus, an information processing method, and an information processing program that can appropriately generate images that reflect multiple correction processes.

[0007] To solve the above problems, an information processing device according to this disclosure comprises: an acquisition unit that acquires a first image that is the subject of a task; a correction processing unit that executes a plurality of correction processes, each being a different correction process, in parallel using the characteristic information of the first image; and a generation unit that generates a third image to be used for processing related to the task using a plurality of second images obtained by each of the plurality of correction processes.

[0008] This figure shows an example of information processing according to the embodiment of this disclosure. This figure shows an example of format conversion processing. This figure shows an example of task adaptation processing. This figure shows an example of the configuration of the information processing apparatus according to the embodiment of this disclosure. This is a flowchart of the information processing procedure according to the embodiment of this disclosure. This is a flowchart of an example of a processing procedure including four types of corrections. This figure shows an example of an adaptation processing result. This figure shows an example of an object detection processing result. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing apparatus.

[0009] Embodiments of this disclosure will be described in detail below with reference to the drawings. Note that these embodiments do not limit the information processing apparatus, information processing method, and information processing program described herein. Furthermore, in each of the following embodiments, the same parts are denoted by the same reference numerals to avoid redundant explanations.

[0010] This disclosure will be described in the following order of items: 1. Embodiments 1-1. Overview of information processing according to the embodiment of this disclosure 1-2. Configuration of the information processing device according to the embodiment 1-3. Procedure of information processing according to the embodiment 1-4. Examples of processing results 1-4-1. Adaptive processing results 1-4-2. Object detection processing results 2. Others 3. Effects of this disclosure 4. Hardware configuration

[0011] <1. Embodiments> <1-1. Overview of Information Processing According to the Embodiments of the Disclosure> First, an overview of the information processing according to the embodiments of the disclosure will be described using Figure 1. Figure 1 is a diagram showing an example of information processing according to the embodiment of the disclosure. Specifically, Figure 1 explains the overview of information processing using object detection as an example. In Figure 1, object detection is explained as an example of a task, but the task is not limited to object detection, but is a task that processes an image as input, and any task to which information processing can be applied is not limited to object detection.

[0012] The information processing according to the embodiments of this disclosure is realized by an information processing device 100 (see Figure 4). Details of the configuration will be described later, but the information processing device 100 is an example of a computer (inference device) that performs processing related to a task. In Figure 1, the information processing device 100 uses an object detection model M1, which is an example of a module whose task is object detection, to take image IN1, which is the image to be detected (task) (also called the "first image") as input and generate image OT1, which is an example of the processing result of object detection (task). For example, the object detection model M1 is an AI (Artificial Intelligence) model (also simply called the "model") generated by a machine learning method.

[0013] Image IN1, the first image in Figure 1, is a RAW (Raw Image Format) image in Bayer array format. Image IN1 is a RAW image detected by a camera (image sensor). Image IN1 may also be a RAW image detected by an image sensor mounted on a moving object such as a car. For example, object detection model M1 is a module (model) trained using machine learning techniques to process the task of object detection. Note that the images in Figure 1, such as image IN1 and component image RI1, do not indicate size, but rather illustrate the processing flow, and their sizes are not limited to those shown in Figure 1. In Figure 1, the entire set of components from the input image IN1 to the output of the processing result (output image) of object detection (task) is described as object detection model M1, but each component included in object detection model M1 may be used individually.

[0014] The following describes the configuration of the object detection model M1 and the processing performed by each component of the object detection model M1. In the example shown in Figure 1, the object detection model M1 includes a format conversion model M11, an adaptive processing model M12, an encoder M13, and a detector M14. Note that the configuration shown in Figure 1 is merely an example, and any configuration can be adopted for the object detection model M1 as long as it can perform multiple correction processes and execute the desired task.

[0015] The format conversion model M11 performs processing related to format conversion. In Figure 1, the format conversion model M11, labeled "Format Converter," is a module (model) that takes a RAW image, image IN1, as input and outputs (generates) an RGB (Red Green Blue) image. In Figure 1, when image IN1 is input, the format conversion model M11 generates an RGB image IG1 from image IN1 that contains three component images: one R (red) component image RI1, one B (blue) component image BI1, and one G (green) component image GI1. In this way, the format conversion model M11 generates a 3-channel image from a 1-channel image.

[0016] For example, the information processing device 100 generates an RGB image IG1 using a format conversion model M11 through the process shown in Figure 2. Figure 2 is a diagram showing an example of the format conversion process. First, the format conversion model M11 generates an RGB image IG0, which is a first RGB image containing four component images: one R (red) component image RI1, one B (blue) component image BI1, and two G (green) component images GI11 and GI12, from the image IN1 (step S11). In this way, the format conversion model M11 generates a four-channel image from a one-channel image.

[0017] For example, the format conversion model M11 generates one R component image RI1, one B component image BI1, and two G component images GI11 and GI12, each half the size of image IN1, from image IN1. Note that the method for generating one R component image, one B component image, and two G component images from a Bayer array RAW image is an existing technology, so a detailed explanation is omitted, but a brief example is provided below.

[0018] For example, the R component image is generated by extracting only the pixels corresponding to red from the RAW image, and concatenating them while maintaining their positional relationships. Similarly, the B component image is generated by extracting only the pixels corresponding to blue from the RAW image, and concatenating them while maintaining their positional relationships. Furthermore, the G component image for the first channel is generated by extracting only the pixels corresponding to the first type (e.g., odd-numbered rows) of green from the RAW image, and concatenating them while maintaining their positional relationships. Similarly, the G component image for the second channel is generated by extracting only the pixels corresponding to green other than the first type (e.g., even-numbered rows) from the RAW image, and concatenating them while maintaining their positional relationships.

[0019] This generates a four-channel RGB image from a one-channel RAW image. Note that the above-described RGB image generation process is merely an example, and the RGB image may be generated by any process as long as the desired RGB image is generated. Through the process described above, the format conversion model M11 converts the one-channel RAW image, image IN1 [W x H x 1], into four RGB images IG0 [W / 2 x H / 2 x 4], each half the size of the RAW image.

[0020] Then, the format conversion model M11 generates RGB image IG1 from RGB image IG0 (step S12). In Figure 1, the format conversion model M11 generates the second RGB image, RGB image IG1, from the first RGB image, RGB image IG0, by converting the two G component images GI11 and GI12 into a single G component image GI1. In this way, the format conversion model M11 generates a 3-channel image from a 4-channel image.

[0021] For example, the format conversion model M11 converts two G-component images into one G-component image through a conversion process as shown in equation (1).

[0022]

[0023] In equation (1), "G" on the left side corresponds to one G component image of the transformed image (second RGB image). In Figure 2, "G" on the left side of equation (1) corresponds to one G component image GI1 of the RGB image IG1. Also, "G" on the right side of equation (1) １ " corresponds to one of the two G component images of the image before conversion (first RGB image). In Figure 2, the "G" on the right side of equation (1) １ " corresponds to the G component image GI11 of one of the RGB images IG0 (for example, the first channel). Also, the "G" on the right side of equation (1) ２ " corresponds to the other of the two G component images of the image before conversion (first RGB image). In Figure 2, the "G" on the right side of equation (1) ２ This corresponds to the G component image GI12 of one of the RGB images IG0 (for example, the second channel).

[0024] For example, the format conversion model M11 generates one G-component image by calculating the average value of each pixel of two G-component images according to formula (1). In FIG. 2, the format conversion model M11 generates one G-component image GI1 by calculating the average value of each pixel of two G-component images GI11 and GI12. Note that the conversion process from the two G-component images described above to one G-component image is just an example. As long as one G-component image is generated, one G-component image may be generated by any process such as a process using a synthesis function that generates (synthesizes) one image from two images.

[0025] In this way, the information processing apparatus 100 converts the RAW image IN1[WxHx1] with a Bayer array (pattern) into a 3-channel RGB image IG1[W / 2xH / 2x3] each having half the size of the RAW image by the process using the format conversion model M11. In the example described above, the information processing apparatus 100 averages the two-channel G-component images to integrate them into a one-channel G-component image, and performs format conversion on the 3-channel image.

[0026] For example, by converting a 4-channel image of RGGB into a 3-channel image of RGB by the above-described process, it becomes possible to appropriately utilize an existing model based on three channels, such as a model pre-learned in an existing RGB image dataset such as ImageNet. In this way, the information processing apparatus 100 can appropriately perform processing as an RGB image by processing the G component as one channel instead of two independent channels. Then, the 3-channel RGB image IG1 converted by the format conversion model M11 is passed to an adaptation processing model M12 that executes an adaptation process (also simply referred to as "adaptation process") for adaptation to a task that is preprocessing in the subsequent stage. Then, an image further optimized corresponding to the task is generated by the adaptation processing model M12, and the image on which this adaptation process is executed is passed to the subsequent encoder M13 and the like, and object detection is performed by the detector M14.

[0027] The adaptation processing model M12 executes adaptation processing for tasks including a plurality of correction processes. In FIG. 1, the adaptation processing model M12 denoted as "Raw Adaptation Module" (RAM) has two main components: adaptive parameterization of the functions of ISP (Image Signal Processing) and feature fusion. As shown in FIG. 3, the adaptation processing model M12 has components that perform various processes. Taking the RGB image IG1 as input, it generates an RGB image IG2 adapted for subsequent object detection through adaptation processing for tasks including a plurality of correction processes. For example, the information processing device 100 generates the RGB image IG2 using the adaptation processing model M12 having a configuration as shown in FIG. 3. FIG. 3 is a diagram showing an example of adaptation processing for a task.

[0028] In the example shown in FIG. 3, the adaptation processing model M12 includes a feature extraction model EM1, a plurality of correction processing models DM1 to DM4, a concatenation processing model CM1, a fusion model FM1, and a normalization model NM1. Note that the configuration shown in FIG. 3 is merely an example, and any configuration can be adopted as long as the adaptation processing model M12 can execute a plurality of correction processes and perform adaptation processing for tasks.

[0029] The feature extraction model EM1 executes processes related to encoding (feature extraction). In FIG. 3, the feature extraction model EM1 denoted as "RPEncoder" (Raw Parameter Encoder) is a module (model) that outputs (generates) the feature amount of the image IN1 as feature information based on the RGB image IG1. The feature extraction model EM1 has a network structure based on CNN (Convolutional Neural Network) and generates information indicating the features of the input image. For example, the feature extraction model EM1 has a network structure that performs convolutional processing and generates information indicating the features of the input image. For example, the feature extraction model EM1 outputs (generates) the feature amount of the image IN1 based on the RGB image IG1 including the R component image RI1, the B component image BI1, and the G component image GI1.

[0030] The correction processing model DM1 performs gamma correction. In FIG. 3, the correction processing model DM1 denoted as "RPDecoder Gamma" (Raw Parameter Decoder Gamma) performs gamma correction using the feature information obtained by the feature extraction model EM1. The correction processing model DM1 estimates the parameters used for gamma correction using the feature information, and performs gamma correction using the estimated parameters corresponding to the gamma correction.

[0031] For example, the correction processing model DM1 performs gamma correction by a correction process as shown in Equation (2). For example, Equation (2) is a correction function corresponding to gamma correction.

[0032]

[0033] The "X" on the right side in Equation (2) corresponds to the input image X. In FIG. 1, the "X" on the right side in Equation (2) corresponds to the image IN1. The "RPD()" on the right side in Equation (2) corresponds to a function that calculates the parameters of gamma correction. The "P(X)" on the right side in Equation (2) corresponds to the feature amount obtained by the feature extraction model EM1. The "θ ｇａｍｍａ " corresponds to the parameter used in the function that calculates the parameters of gamma correction. The "X ｇａｍｍａ " on the left side in Equation (2) corresponds to the second image that is the image obtained by performing the gamma correction process on the input image X.

[0034] For example, the correction processing model DM1 performs gamma correction on the input image X using the gamma correction parameter RPD(P(X); θ ｇａｍｍａ ) estimated based on the feature amount P(X) obtained by the feature extraction model EM1. For example, the correction processing model DM1 performs gamma correction on the input image X by an operation with the gamma correction parameter RPD(P(X); θ ｇａｍｍａ ) as the exponent. Thereby, the correction processing model DM1 generates "X ｇａｍｍａ ", which is the processed image on which the gamma correction has been performed. For example, the processed image "X ｇａｍｍａ " is an RGB three-channel image.

[0035] The correction processing model DM2 performs color correction. In Figure 3, the correction processing model DM2, labeled "RPDecoder CCM" (Raw Parameter Decoder Computer Color Matching), performs color correction using the feature information obtained by the feature extraction model EM1. The correction processing model DM2 estimates the parameters to be used for color correction using the feature information, and then performs color correction using the estimated parameters corresponding to the color correction.

[0036] For example, the correction processing model DM2 performs color correction by a correction process as shown in equation (3). For example, equation (3) is a correction function corresponding to color correction.

[0037]

[0038] In equation (3), "R" on the right-hand side corresponds to the R component of the input image X. In Figure 1, "R" on the right-hand side of equation (3) corresponds to the R component of image IN1. In equation (3), "B" on the right-hand side corresponds to the B component of the input image X. In Figure 1, "B" on the right-hand side of equation (3) corresponds to the B component of image IN1. In equation (3), "G" on the right-hand side corresponds to the G component of the input image X. In Figure 1, "G" on the right-hand side of equation (3) corresponds to the G component of image IN1. １１ "~"c ３３ The nine elements of " correspond to the color correction parameters. In equation (3), the left side is "X ｃｃｍ This corresponds to the second image, which is obtained by performing color correction processing on the input image X.

[0039] For example, the correction processing model DM2 applies the following to the input image X: １１ "~"c ３３ Color correction is performed using the following nine parameters. For example, the correction processing model DM2 is calculated using equation (3) for "c １１ "~"c ３３ The input image X is color corrected by a matrix operation using the nine parameters of " and each RGB component of the input image X as elements. As a result, the correction processing model DM2 produces the processed image "X" which has undergone color correction. ｃｃｍ It generates "X". For example, the processed image is "Xｃｃｍ This is an image with three RGB channels.

[0040] The correction processing model DM3 performs white balance correction. In Figure 3, the correction processing model DM3, labeled "RPDecoder WB" (Raw Parameter Decoder White Balance), performs white balance correction using the feature information obtained by the feature extraction model EM1. The correction processing model DM3 uses the feature information to estimate the parameters to be used for white balance correction, and then performs white balance correction using the estimated parameters corresponding to the white balance correction.

[0041] For example, the correction processing model DM3 performs white balance correction by the correction processing shown in equation (4). For example, equation (4) is a correction function corresponding to white balance correction.

[0042]

[0043] In equation (4), "R" on the right-hand side corresponds to the R component of the input image X. In Figure 1, "R" on the right-hand side of equation (4) corresponds to the R component of image IN1. In equation (4), "B" on the right-hand side corresponds to the B component of the input image X. In Figure 1, "B" on the right-hand side of equation (4) corresponds to the B component of image IN1. In equation (4), "G" on the right-hand side corresponds to the G component of the input image X. In Figure 1, "G" on the right-hand side of equation (4) corresponds to the G component of image IN1.

[0044] The right-hand side of equation (4) is "α Ｒ " corresponds to the white balance correction parameter for the R component. The "α" on the right side of equation (4) Ｇ " corresponds to the white balance correction parameter for the G component. The "α" on the right side of equation (4) Ｂ The "X" on the left side of equation (4) corresponds to the white balance correction parameter for the B component. The operator containing a point in the circle on the right side of equation (4) corresponds to the Hadamard product operator (Hadamard operator). For example, the operator containing a point in the circle on the right side of equation (4) means element-wise multiplication. ｗｂThis corresponds to the second image, which is obtained by performing white balance correction processing on the input image X.

[0045] For example, the correction processing model DM3 applies the following to the input image X: Ｒ "α Ｇ "α Ｂ White balance correction is performed using the following three parameters: For example, the correction processing model DM3 uses equation (4) to determine "α Ｒ Multiply the R component of the input image X by "α Ｇ Multiply the input image X by the G component of "α Ｂ White balance correction is performed on the input image X by multiplying the B component of the input image X by the calculation ". For example, the correction processing model DM3 performs white balance correction on the input image X by performing the process of equation (4) on all pixels. As a result, the correction processing model DM3 obtains the processed image "X" which has had white balance correction performed. ｗｂ It generates "X". For example, the processed image is "X ｗｂ This is an image with three RGB channels.

[0046] The correction processing model DM4 performs brightness correction. In Figure 3, the correction processing model DM4, labeled "RPDecoder Brightness" (Raw Parameter Decoder Brightness), performs brightness correction using the feature information obtained by the feature extraction model EM1. The correction processing model DM4 estimates the parameters to be used for brightness correction using the feature information, and then performs brightness correction using the estimated parameters corresponding to the brightness correction.

[0047] For example, the correction processing model DM4 performs brightness correction by a correction process as shown in equation (5). For example, equation (5) is a correction function corresponding to brightness correction.

[0048]

[0049] In equation (5), "X" on the right side corresponds to the input image X. In Figure 1, "X" on the right side of equation (5) corresponds to image IN1. "RPD()" on the right side of equation (5) corresponds to the function that calculates the brightness correction parameters. "P(X)" on the right side of equation (5) corresponds to the features obtained by the feature extraction model EM1. "θbrightness" on the right side of equation (5) corresponds to the parameters used in the function that calculates the brightness correction parameters. ｂｒｉｇｈｔｎｅｓｓ This corresponds to the second image, which is obtained by performing brightness correction processing on the input image X.

[0050] For example, the correction processing model DM4 performs brightness correction on the input image X using a brightness correction parameter RPD(P(X); θbrightness) estimated based on the feature quantity P(X) obtained by the feature extraction model EM1. For example, the correction processing model DM4 performs brightness correction on the input image X by adding the brightness correction parameter RPD(P(X); θbrightness) according to equation (5).

[0051] Thus, the correction processing model DM4 calculates the brightness correction parameter RPD(P(X); θbrightness), which is the bias component of brightness correction, based on the features obtained from the input image X, and performs brightness correction on the input image X according to the value of the calculated parameter. As a result, the correction processing model DM4 produces a processed image "X" on which brightness correction has been performed. ｂｒｉｇｈｔｎｅｓｓ It generates "X". For example, the processed image is "X ｂｒｉｇｈｔｎｅｓｓ This is an RGB 3-channel image. In the following, when correction processing models DM1 to DM4 are not distinguished, they will be referred to as "correction processing model DM".

[0052] Furthermore, the correction processing model DM is not limited to correction processing models DM1 to DM4, and may include models that perform correction processing other than gamma correction, color correction, white balance correction, and brightness correction. In other words, the correction processing is not limited to gamma correction, color correction, white balance correction, and brightness correction, but may include any correction processing. For example, the correction processing is not limited to gamma correction, color correction, white balance correction, and brightness correction, but may include various correction processing such as sharpness correction and contrast correction.

[0053] The concatenation processing model CM1 performs a concatenation process that combines the information after correction processing. In Figure 3, the concatenation processing model CM1, labeled "CONCAT," is a single concatenation module for integrating information from representations that have undergone various processes. For example, the adaptive processing model M12 has a reverse hourglass configuration.

[0054] For example, the concatenation processing model CM1 concatenates all the processed images obtained from each correction process into a single multi-channel tensor by a concatenation process as shown in equation (6).

[0055]

[0056] In equation (6), the left side is "X ｆｕｓｅｄ " corresponds to the concatenated image (tensor) obtained by the concatenation process. The "X" on the right side of equation (6) ｇａｍｍａ " corresponds to the processed image in which gamma correction has been performed by the correction processing model DM1. The "X" on the right side of equation (6) ｃｃｍ " corresponds to the processed image in which color correction has been performed by the correction processing model DM2. The "X" on the right side of equation (6) ｗｂ " corresponds to the processed image in which white balance correction processing has been performed by the correction processing model DM3. The "X" on the right side of equation (6) ｂｒｉｇｈｔｎｅｓｓ This corresponds to a processed image in which brightness correction processing has been performed by the correction processing model DM4.

[0057] For example, the concatenation processing model CM1 generates a tensor (also called a "concatenated image") by concatenating the gamma-corrected processed image, the color-corrected processed image, the white-balance-corrected processed image, and the brightness-corrected processed image. For example, the concatenation processing model CM1 generates a 12-channel image (concatenated image) by concatenating the gamma-corrected processed image, the color-corrected processed image, the white-balance-corrected processed image, and the brightness-corrected processed image, each of which has 3 channels, using equation (6).

[0058] The fusion model FM1 generates a third image, which is an image created by fusing the images contained in the concatenated image obtained by the concatenation processing model CM1. In Figure 3, the fusion model FM1, labeled "Fusion Module," is a module (model) that outputs (generates) a 3-channel image from a 12-channel image (concatenated image). For example, the fusion model FM1 converts the concatenated image obtained by the concatenation processing model CM1 into an image to be used in subsequent processing.

[0059] The fusion model FM1 has a CNN-based network structure and generates an image (third image) with a compressed number of channels in the input concatenated image. For example, the fusion model FM1 has a network structure that performs convolution processing and compresses a 12-channel image (conaten image) into a 3-channel image (third image). For example, the fusion model FM1 outputs (generates) a 3-channel RGB image (third image) containing one R component image, one B component image, and one G component image from a 12-channel RGB image (conaten image) containing four R component images, four B component images, and four G component images.

[0060] Based on the processing of the concatenation model CM1 and the fusion model FM1 described above, the adaptive processing model M12 generates a third image by fusing multiple second images. The fusion model FM1 is not limited to generating a third image based on CNN convolution processing, but may generate a third image by any processing. The fusion model FM1 may also generate a third image by calculating the average of multiple second images. For example, the fusion model FM1 generates one R component image based on the average of four R component images from a 12-channel image. For example, the fusion model FM1 calculates the average pixel value of each pixel using the pixel values ​​of each pixel in the four R component images in a 12-channel image, and generates one R component image using the calculated average pixel value of each pixel. Similarly, the fusion model FM1 generates one B component image based on the average of four B component images from a 12-channel image. Furthermore, the fusion model FM1 generates one G component image based on the average of four G component images from a 12-channel image. For example, the fusion model FM1 may generate a third image based on set parameters. In this case, the fusion model FM1 may generate the third image using 12 channels of images and parameters (e.g., weights) corresponding to each of those images. For example, the fusion model FM1 may generate the third image by performing calculations using 12 channels of images weighted by parameters.

[0061] The normalization model NM1 performs normalization processing on the third image obtained by the fusion model FM1. In Figure 3, the normalization model NM1, labeled "Normalization," is a module (model) that takes the third image as input and outputs the third image after normalization processing. For example, the normalization model NM1 performs normalization processing so that the pixel values ​​of each pixel in the third image fall within a predetermined range. The normalization processing is the same as existing processing, so a detailed explanation is omitted, but for example, the normalization model NM1 performs normalization processing so that the pixel values ​​of each pixel in the third image fall within a range between a predetermined minimum value and a predetermined maximum value.

[0062] Through the process described above, the adaptive processing model M12 generates a third image, an RGB image IG2, which includes an adaptive processing that includes multiple correction processes, in response to the input of an RGB image IG1, which includes an R component image RI1, a B component image BI1, and a G component image GI1. This RGB image IG2 includes an R component image RI2, a B component image BI2, and a G component image GI2.

[0063] Encoder M13 performs encoding (feature extraction) processing for object detection. Encoder M13, labeled "Encoder" in Figure 1, is a module (model) that outputs (generates) feature quantities of image IN1 as feature information based on the third image, the RGB image IG2. Encoder M13 has a CNN-based network structure and generates information that indicates the features of the input image. For example, Encoder M13 has a network structure that performs convolution processing and generates information that indicates the features of the input image. For example, Encoder M13 outputs (generates) feature quantities of image IN1 based on the RGB image IG2, which includes the R component image RI2, the B component image BI2, and the G component image GI2.

[0064] Detector M14 performs object detection processing. In Figure 1, Detector M14, labeled "Detector," is a module (model) that performs object detection processing using the features of image IN1 extracted based on the third image, the RGB image IG2, and outputs (generates) the processing results. For example, Detector M14 performs object detection processing using the features extracted from the image. Note that Detector M14 can employ any network configuration as long as it enables the desired object detection, and may use a network configuration similar to that of conventional object detection detectors. In Figure 1, Detector M14 outputs image OT1 showing the processing results of object detection (task) by performing object detection processing using the features obtained by encoder M13.

[0065] In the process described above, a concatenated image, formed by combining the processed images of multiple correction processing models DM, is passed to the fusion module, the fusion model FM1. The fusion model FM1 then combines the processed images from different ISP functions to generate (synthesize) an integrated and optimized 3-channel representation with the same spatial dimensions as the original input. After processing by the fusion model FM1 (fusion step), batch normalization is applied by the normalization model NM1. For example, in object detection, mean-standard deviation normalization is generally applied to the input image, and processing by the normalization model NM1 can standardize the data and improve the convergence of the model. If the mean and standard deviation of the data cannot be calculated in advance, batch normalization by the normalization model NM1 may be used to adaptively adjust the process. This allows appropriate normalization parameters to be learned during training. The output normalized by the normalization model NM1 is then passed as input to the encoder M13. This integration allows the information processing device 100 to jointly optimize ISP parameters and object detection tasks by integrating the adaptive processing model M12 into the object detection pipeline, thereby creating an input representation suitable for the data and improving overall performance.

[0066] As described above, the information processing device 100 generates a second image corresponding to each correction process using multiple correction process models DM, and generates a third image by combining (merging) multiple second images. This allows the information processing device 100 to appropriately generate an image that reflects multiple correction processes.

[0067] For example, conventional object detectors are trained and evaluated using RGB images processed with predefined image signal processing (ISP) parameters. The adaptive processing model M12 allows for a more flexible approach that adapts to the input image. For instance, the adaptive processing model M12 is designed to learn to optimize ISP parameters along with the object detection task, determining appropriate parameters (weights) for each ISP process and discarding processes that are unnecessary or degrading in performance. Furthermore, the adaptive processing model M12 is designed to be easily integrated into any object detection pipeline and to be trained end-to-end.

[0068] In the object detection model M1 shown in Figure 1, the format of the RAW image is converted from a Bayer pattern to a 3-channel RGB image format. The adaptive processing model M12 of the object detection model M1 then converts the 3-channel RGB image into image features. The adaptive processing model M12 then estimates parameters for gamma correction, color correction, white balance correction, and brightness correction using the converted features, performs individual image processing using these parameters, and after combining these four processed images, performs a synthesis process to output a 3-channel color image.

[0069] For example, the adaptive processing model M12 is responsible for adaptively predicting and applying ISP parameters. For example, the adaptive processing model M12 uses a parameter encoder (RPE) and a parameter decoder (RPD) for each correction process (also called an "ISP function"). For example, the RPE is a convolution-based encoder that receives a RAW image and generates an encoding vector. For example, the RPD receives image features and estimates (predicts) the parameters of the corresponding ISP function.

[0070] For example, in object detection model M1, a single RPE is applied to the RAW image, and the generated features are shared with the independent RPDs of each function. This approach is based on the observation that many ISP functions can leverage similar input features, and it is estimated that there is no need to extract separate features for each function. Adaptive processing model M12 applies the predicted parameters to the input image in parallel. This parallel architecture is important not only for efficiency but also to ensure that all potential features are available to the subsequent fusion module (e.g., fusion model FM1). The fusion module performs effective filtering, selectively highlighting the features most relevant to the object detection task and excluding others. The ISP functions incorporated into adaptive processing model M12 may be empirically selected from existing ISP functions that are particularly relevant to performance. For example, ISP functions may include gamma correction for tone mapping.

[0071] <1-2. Configuration of an Information Processing Device According to an Embodiment> Next, the configuration of an information processing device 100, which is an example of an information processing device that performs information processing according to an embodiment, will be described. Figure 4 is a diagram showing an example of the configuration of an information processing device according to an embodiment of this disclosure. For example, the information processing device 100 shown in Figure 4 is an example of an information processing device.

[0072] As shown in Figure 4, the information processing device 100 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15. In the example in Figure 4, the information processing device 100 includes an input unit 12 (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and an output unit 13 (for example, a liquid crystal display) that outputs (displays, etc.) various information.

[0073] The communication unit 11 is implemented, for example, by a NIC (Network Interface Card) or a communication circuit. The communication unit 11 is connected to a communication network N (such as the Internet) by wire or wireless connection and transmits and receives information with other devices via the communication network N.

[0074] The input unit 12 receives various operations from the user. The input unit 12 accepts input from the user. The input unit 12 accepts input from the user of information used to generate a mesh model. The input unit 12 may also accept various operations from the user via a keyboard, mouse, or touch panel provided on the information processing device 100.

[0075] The output unit 13 outputs various information. The output unit 13 has a display device (display unit) such as a display and displays various information. The output unit 13 displays information generated by the generation unit 155. The output unit 13 displays processing results using the object detection model M1. The output unit 13 displays information related to processing by the task processing unit 156. The output unit 13 displays the processing results of object detection by the task processing unit 156.

[0076] Furthermore, the output unit 13 may have functions to output information in various ways, not limited to display functions. For example, the output unit 13 may have a function to output information as sound. For example, the output unit 13 may have an audio output unit such as a speaker that outputs sound.

[0077] The storage unit 14 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The storage unit 14 has a model information storage unit 141.

[0078] The model information storage unit 141 according to this embodiment stores information about a model. For example, the model information storage unit 141 stores information (model data) that shows the structure of a model (network). The model information storage unit 141 includes items such as "model ID," "purpose," and "model data."

[0079] The "Model ID" indicates identification information for identifying the model. The "Purpose" indicates the purpose of the corresponding model. The "Model Data" indicates the data of the model. For example, the "Model Data" includes various pieces of information that make up the model, such as information about the network included in the model and functions.

[0080] The model ID "M1" that identifies the object detection model M1 is associated with information indicating that its purpose is "object detection." This indicates that the object detection model M1 is a model used for the task of object detection. In addition, the model ID "M1" that identifies the object detection model M1 is associated with information indicating that the model data for the object detection model M1 is model data MDT1.

[0081] Furthermore, the model ID "M1" that identifies the object detection model M1 is associated with and stored the model IDs that identify the models included in the object detection model M1. For example, the model ID "M1" is associated with and stored the model IDs "M11", "M12", "M13", "M14", etc., which identify the models included in the object detection model M1.

[0082] The model ID "M11" that identifies the format conversion model M11 is associated with information indicating that it is a model included in the object detection model M1 and that its purpose is "format conversion". This indicates that the format conversion model M11 is a model used for format conversion. In addition, the model ID "M11" that identifies the format conversion model M11 is associated with information indicating that the model data of the format conversion model M11 is model data MDT11.

[0083] The model ID "M12" that identifies the adaptive processing model M12 is associated with and stored information indicating that it is a model included in the object detection model M1 and that its purpose is "adaptive processing". This indicates that the adaptive processing model M12 is a model used for adaptive processing. In addition, the model ID "M12" that identifies the adaptive processing model M12 is associated with and stored information indicating that the model data of the adaptive processing model M12 is model data MDT12.

[0084] Furthermore, the model ID "M12" that identifies the adaptive processing model M12 is associated with and stored the model IDs that identify the models included in the adaptive processing model M12. For example, the model ID "M12" is associated with and stored the model IDs "EM1", "DM1", "DM2", "DM3", "DM4", "CM1", "FM1", "NM1", etc., which identify the models included in the adaptive processing model M12.

[0085] The model ID "EM1" that identifies the feature extraction model EM1 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "feature extraction for correction processing." This indicates that the feature extraction model EM1 is a model used for feature extraction for correction processing. In addition, the model ID "EM1" that identifies the feature extraction model EM1 is associated with information indicating that the model data for the feature extraction model EM1 is model data EMDT1.

[0086] The model ID "DM1" that identifies the correction processing model DM1 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "gamma correction". This indicates that the correction processing model DM1 is a model used for gamma correction. In addition, the model ID "DM1" that identifies the correction processing model DM1 is associated with information indicating that the model data for the correction processing model DM1 is model data DMDT1.

[0087] The model ID "DM2" that identifies the correction processing model DM2 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "color correction". This indicates that the correction processing model DM2 is a model used for color correction. In addition, the model ID "DM2" that identifies the correction processing model DM2 is associated with information indicating that the model data for the correction processing model DM2 is model data DMDT2.

[0088] The model ID "DM3" that identifies the correction processing model DM3 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "white balance correction." This indicates that the correction processing model DM3 is a model used for white balance correction. In addition, the model ID "DM3" that identifies the correction processing model DM3 is associated with information indicating that the model data for the correction processing model DM3 is model data DMDT3.

[0089] The model ID "DM4" that identifies the correction processing model DM4 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "brightness correction". This indicates that the correction processing model DM4 is a model used for brightness correction. In addition, the model ID "DM4" that identifies the correction processing model DM4 is associated with information indicating that the model data for the correction processing model DM4 is model data DMDT4.

[0090] The model ID "CM1" that identifies the linking processing model CM1 is associated with and stores information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "linking processing". Linking processing model CM1 indicates that it is a model used for linking processing. In addition, the model ID "CM1" that identifies the linking processing model CM1 is associated with and stores information indicating that the model data of linking processing model CM1 is model data CMDT1.

[0091] The model ID "FM1" that identifies the fusion model FM1 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "synthesis processing". This indicates that the fusion model FM1 is a model used for synthesis processing. In addition, the model ID "FM1" that identifies the fusion model FM1 is associated with information indicating that the model data of the fusion model FM1 is model data FMDT1.

[0092] The model ID "NM1" that identifies the normalization model NM1 is associated with information indicating that it is a model included in the adaptive processing model M12 and that its purpose is "normalization processing". The normalization model NM1 indicates that it is a model used for normalization processing. Furthermore, the model ID "NM1" that identifies the normalization model NM1 is associated with information indicating that the model data for normalization model NM1 is model data NMDT1.

[0093] The model ID "M13" that identifies encoder M13 is associated with information indicating that it is a model included in object detection model M1 and that its purpose is "feature extraction for a task." This indicates that encoder M13 is a model used for feature extraction for a task. In addition, the model ID "M13" that identifies encoder M13 is associated with information indicating that the model data for encoder M13 is model data MDT13.

[0094] The model ID "M14" that identifies detector M14 is associated with information indicating that it is a model included in object detection model M1 and that its purpose is "object detection." This indicates that detector M14 is a model used for object detection. In addition, the model ID "M14" that identifies detector M14 is associated with information indicating that the model data for detector M14 is model data MDT14.

[0095] The model information storage unit 141 is not limited to the above and may store various types of information depending on the purpose. For example, the model information storage unit 141 stores parameter information of a model learned (generated) through a learning process.

[0096] Furthermore, the memory unit 14 stores various other types of information besides those mentioned above. For example, the memory unit 14 stores various types of information used in processing. For example, the memory unit 14 stores images used in processing. For example, the memory unit 14 stores images that are the target of a task. For example, the memory unit 14 stores correction functions used in processing. For example, the memory unit 14 stores information generated by the generation unit 155. The memory unit 14 stores images generated by the generation unit 155.

[0097] The control unit 15 is implemented, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program (for example, the information processing program according to this disclosure) stored inside the information processing device 100 using RAM (Random Access Memory) or the like as a working area. The control unit 15 is also a controller and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0098] As shown in Figure 4, the control unit 15 includes an acquisition unit 151, a format conversion unit 152, a feature extraction unit 153, a correction processing unit 154, a generation unit 155, a task processing unit 156, and a transmission unit 157, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in Figure 4, and other configurations are also possible as long as they perform the information processing described later. For example, among the control unit 15, the format conversion unit 152, the feature extraction unit 153, the correction processing unit 154, the generation unit 155, and the task processing unit 156 may be configured as a single processing unit. In this case, the processing unit executes processing related to the task using a task processing model such as the object detection model M1.

[0099] The acquisition unit 151 acquires various types of information. The acquisition unit 151 acquires various types of information from an external information processing device. The acquisition unit 151 acquires various types of information from the storage unit 14. The acquisition unit 151 acquires information received by the input unit 12. The acquisition unit 151 acquires images received by the input unit 12.

[0100] The acquisition unit 151 acquires the object detection model M1 from the model information storage unit 141. The acquisition unit 151 acquires an image from the storage unit 14. The acquisition unit 151 acquires the first image that is the target of the task. The acquisition unit 151 receives the first image via the communication unit 11. The acquisition unit 151 receives the first image from the requesting device that requests processing of the task.

[0101] The format conversion unit 152 performs processing related to format conversion. The format conversion unit 152 performs processing related to format conversion using the information stored in the storage unit 14.

[0102] The format conversion unit 152 generates an RGB image from a first image, which is a RAW image. The format conversion unit 152 generates an RGB image based on two G component images, one R component image, and one B component image obtained from the first image, which is a Bayer image.

[0103] The format conversion unit 152 generates a first RGB image from the first image, which is a Bayer image, which includes two G component images, one R component image, and one B component image. From the first RGB image, it combines two G component images into one G component image to generate a second RGB image that includes the combined one G component image, one R component image, and one B component image. The format conversion unit 152 generates a first RGB image from the first image, which is a Bayer image, which includes two G component images, one R component image, and one B component image that are half the size of the first image. It then generates a single G component image by calculating the average value of each pixel of the two G component images.

[0104] For example, the format conversion unit 152 uses a module related to format conversion within the object detection model M1 to perform format conversion processing. For example, the format conversion unit 152 uses a format conversion model M11 to perform format conversion processing.

[0105] The feature extraction unit 153 performs processing related to feature extraction. The feature extraction unit 153 performs processing related to feature extraction using the information stored in the storage unit 14. The feature extraction unit 153 performs processing related to feature extraction using the processing result of the format conversion unit 152.

[0106] The feature extraction unit 153 extracts feature information from the first image. The feature extraction unit 153 extracts feature information from the first image using the RGB image. The feature extraction unit 153 extracts feature information from the first image using the second RGB image. The feature extraction unit 153 extracts feature information from the first image based on CNN.

[0107] For example, the feature extraction unit 153 executes feature extraction processing using the feature extraction module of the object detection model M1. For example, the feature extraction unit 153 executes feature extraction processing using the feature extraction module of the adaptive processing model M12. For example, the feature extraction unit 153 executes feature extraction processing using the feature extraction model EM1.

[0108] The correction processing unit 154 performs correction-related processing. The correction processing unit 154 performs correction-related processing using the information stored in the storage unit 14. The correction processing unit 154 performs correction-related processing using the processing results of the feature extraction unit 153.

[0109] The correction processing unit 154 uses the feature information of the first image to execute multiple correction processes in parallel, each of which is a different correction process. The correction processing unit 154 executes multiple correction processes including gamma correction. The correction processing unit 154 executes multiple correction processes including color correction. The correction processing unit 154 executes multiple correction processes including white balance correction. The correction processing unit 154 executes multiple correction processes including brightness correction.

[0110] The correction processing unit 154 performs multiple correction processes using the feature information extracted from the first image by the feature extraction unit 153. The correction processing unit 154 estimates the parameters to be used for each of the multiple correction processes using the feature information, and generates multiple second images by executing each correction process using the estimated parameters corresponding to each correction process. The correction processing unit 154 generates multiple second images by executing each correction process using the parameters corresponding to each correction process and the correction function corresponding to each correction process. The correction processing unit 154 generates multiple second images, which are multiple RGB images in which each of the multiple correction processes has been executed.

[0111] For example, the correction processing unit 154 executes correction processing using the correction processing module of the object detection model M1. For example, the correction processing unit 154 executes correction processing using the correction processing module of the adaptive processing model M12. For example, the correction processing unit 154 executes correction processing using the correction processing model DM.

[0112] The correction processing unit 154 performs gamma correction processing using correction processing model DM1. The correction processing unit 154 performs color correction processing using correction processing model DM2. The correction processing unit 154 performs white balance correction processing using correction processing model DM3. The correction processing unit 154 performs brightness correction processing using correction processing model DM4.

[0113] The generation unit 155 executes processing related to the generation process. The generation unit 155 executes the generation process using the information stored in the storage unit 14. The generation unit 155 executes the generation process using the processing result of the correction processing unit 154.

[0114] The generation unit 155 generates a third image to be used for processing related to the task using a plurality of second images obtained by each of a plurality of correction processes. The generation unit 155 generates a third image using a plurality of second images, including a second image on which gamma correction has been performed. The generation unit 155 generates a third image using a plurality of second images, including a second image on which color correction has been performed. The generation unit 155 generates a third image using a plurality of second images, including a second image on which white balance correction has been performed. The generation unit 155 generates a third image using a plurality of second images, including a second image on which brightness correction has been performed.

[0115] The generation unit 155 generates a third image by fusing multiple second images. The generation unit 155 generates a third image based on parameters calculated from multiple second images by CNN. The generation unit 155 generates a third image based on set parameters. The generation unit 155 generates a third image by calculating the average of multiple second images. The generation unit 155 generates a third image by averaging multiple second images. The generation unit 155 generates a third image by averaging multiple second images for each corresponding pixel. The generation unit 155 calculates an average value for each corresponding pixel of multiple second images and generates a third image using the calculated average value as the pixel value for that pixel.

[0116] For example, the generation unit 155 executes generation processing using a module related to generation processing from the object detection model M1. For example, the generation unit 155 executes generation processing using a module related to generation processing from the adaptive processing model M12. For example, the generation unit 155 executes generation processing using the concatenation processing model CM1 and the fusion model FM1. In addition, the generation unit 155 executes normalization processing using the normalization model NM1.

[0117] The generation unit 155 may generate various types of information. The generation unit 155 may generate various types of information to be displayed. For example, the generation unit 155 may generate content to be provided to the user. In this case, the generation unit 155 generates screen-related information (images) using various conventional technologies related to images as appropriate. The generation unit 155 generates images using various conventional technologies related to GUI (Graphical User Interface) as appropriate. For example, the generation unit 155 may generate images using CSS (Cascading Style Sheets), JavaScript (registered trademark), HTML (Hyper Text Markup Language), or any language capable of describing the information processing such as the information display and operation reception described above.

[0118] The task processing unit 156 executes processing related to the task. The task processing unit 156 executes processing related to the task using the information stored in the storage unit 14. The task processing unit 156 executes processing related to the task using the processing results of the generation unit 155.

[0119] For example, the task processing unit 156 executes task-related processing using the task processing module of the object detection model M1. For example, the task processing unit 156 executes task-related processing using the encoder M13 and detector M14. For example, the task processing unit 156 performs object detection using the encoder M13 and detector M14.

[0120] The transmitting unit 157 transmits various information. The transmitting unit 157 provides various information. The transmitting unit 157 provides various information to an external information processing device. The transmitting unit 157 transmits various information to an external information processing device. The transmitting unit 157 transmits information stored in the storage unit 14. The transmitting unit 157 transmits information stored in the model information storage unit 141.

[0121] The transmitting unit 157 transmits the information generated by the generation unit 155. The transmitting unit 157 transmits the image generated by the generation unit 155 to an external device. The transmitting unit 157 transmits information related to the processing performed by the task processing unit 156. The transmitting unit 157 transmits information related to the processing performed by the task processing unit 156 to an external device. The transmitting unit 157 transmits the object detection processing results from the task processing unit 156 to an external device.

[0122] <1-3. Information Processing Procedure According to the Embodiment> The information processing procedure according to the embodiment will be described below. First, the information processing procedure according to the embodiment will be described using Figure 5. Figure 5 is a flowchart of the information processing procedure according to the embodiment of this disclosure.

[0123] As shown in Figure 5, the information processing device 100 acquires a first image that is the target of the task (step S101). For example, the information processing device 100 acquires a RAW image that is the target of object detection as the first image.

[0124] Then, the information processing device 100 generates an RGB image from the first image (step S102). For example, the information processing device 100 generates an RGB image from a RAW image.

[0125] Then, the information processing device 100 extracts feature information of the first image using the RGB image (step S103). For example, the information processing device 100 extracts feature information that is commonly used in multiple correction processes using the RGB image.

[0126] Then, the information processing device 100 uses the feature information of the first image to perform multiple correction processes, each of which is a different correction process (step S104). For example, the information processing device 100 uses the feature information of the first image to perform multiple correction processes and generate multiple second images.

[0127] Then, the information processing device 100 generates a third image to be used for processing related to the task using the multiple second images obtained by each of the multiple correction processes (step S105). For example, the information processing device 100 generates a third image to be used for object detection processing using the multiple second images generated by each of the multiple correction processes.

[0128] Then, the information processing device 100 performs task processing using the third image (step S106). For example, the information processing device 100 performs object detection processing using the third image, which has been modified by multiple correction processes.

[0129] Next, using Figure 6, we will explain an example of a processing procedure when there are four types of correction processes: gamma correction, color correction, white balance correction, and brightness correction. Figure 6 is a flowchart of an example of a processing procedure that includes four types of correction. Specifically, Figure 6 is a flowchart of an example of a processing procedure that includes gamma correction, color correction, white balance correction, and brightness correction. Note that explanations of points similar to those in Figure 5 will be omitted as appropriate. For example, the processing in steps S201 to S203 in Figure 6 is the same as the processing in steps S101 to S103 in Figure 5, so a detailed explanation will be omitted.

[0130] As shown in Figure 6, the information processing device 100 acquires a first image that is the target of the task (step S201). Then, the information processing device 100 generates an RGB image from the first image (step S202). Then, the information processing device 100 extracts feature information of the first image using the RGB image (step S203).

[0131] Then, the information processing device 100 performs gamma correction processing using the feature information of the first image (step S204-1). The information processing device 100 also performs color correction processing using the feature information of the first image (step S204-2). The information processing device 100 also performs white balance correction processing using the feature information of the first image (step S204-3). The information processing device 100 also performs brightness correction processing using the feature information of the first image (step S204-4). Note that the processes in steps S204-1 to S204-4 may be started simultaneously or at different timings, provided that they are processed before step S205.

[0132] Then, the information processing device 100 generates a third image to be used for processing related to the task using a plurality of second images obtained by each of the plurality of correction processes (step S205). For example, the information processing device 100 generates a third image to be used for object detection processing using a plurality of second images generated by gamma correction, color correction, white balance correction, and brightness correction, respectively.

[0133] Then, the information processing device 100 performs task processing using the third image (step S206). For example, the information processing device 100 performs object detection processing using the third image, which has been corrected by gamma correction, color correction, white balance correction, and brightness correction.

[0134] <1-4. Examples of Processing Results> Below, we will explain some examples of the results obtained by the above-described processing, based on the diagrams.

[0135] <1-4-1. Adaptive Processing Results> First, Figure 7 will be used to explain the results of adaptive processing by adaptive processing model M12, which includes multiple correction processes. Figure 7 is a diagram showing an example of adaptive processing results. Specifically, Figure 7 is a diagram showing an example of an image before adaptive processing and an image after adaptive processing.

[0136] The image labeled "Pre-processing Image" on the left side of Figure 7 (also referred to as "Pre-processing Image") shows the image before adaptive processing by the adaptive processing model M12. For example, the pre-processing image in Figure 7 corresponds to the first image. In the case of Figure 1, the RGB image IG1 corresponds to the pre-processing image in Figure 7.

[0137] The image labeled "Processed Image" on the right side of Figure 7 (also referred to as "Processed Image") shows the image after adaptive processing by the adaptive processing model M12. For example, the processed image in Figure 7 corresponds to the third image. In the processing example explained in Figure 1, the RGB image IG2 corresponds to the processed image in Figure 7.

[0138] <1-4-2. Object Detection Processing Results> Next, the object detection processing results by the object detection model M1 will be explained using Figure 8. Figure 8 is a diagram showing an example of the object detection processing results. Specifically, Figure 8 is a diagram showing an example of the object detection results.

[0139] The image labeled "Ground-truth: Correct Answer" on the right side of Figure 8 (also called the "correct answer image") shows the correct answer when object detection is performed using that image. For example, the rectangular frame in the correct answer image of Figure 8 shows the correct answer for the object detection task when object detection is performed using that image.

[0140] The image labeled "Object Detection Results" on the left side of Figure 8 (also referred to as "Object Detection Results") shows the processing result of the object detection task performed by the object detection model M1. For example, the object detection results in Figure 8 correspond to the processing result of the object detection (task). In the case of Figure 1, image OT1 corresponds to the object detection results in Figure 8.

[0141] As shown in the object detection results in Figure 8, object detection using the object detection model M1 can appropriately detect the correct object shown in the ground truth image. As shown in the processing results above, in the example in Figure 7, the unprocessed image may be a more appropriate image than the processed image when viewed by a human. On the other hand, when processing a task such as object detection, as shown in Figure 8, the processed image is a more appropriate image because object detection has been performed appropriately. In this way, the information processing, including multiple correction processes, performed by the information processing device 100 enables the appropriate execution of the object detection task.

[0142] <2. Others> The above-described process is merely an example, and the information processing device 100 may perform various processes. For example, the information processing device 100 may perform a learning process to learn various AI models such as the object detection model M1. In this case, the control unit 15 of the information processing device 100 may include a learning unit that performs the learning process. Also, for example, the storage unit 14 of the information processing device 100 may store data used for the learning process (learning data).

[0143] The learning unit of the information processing device 100 (specifically the control unit 15) executes a learning process to learn various models. The learning unit learns various information based on information from an external information processing device and information stored in the storage unit 14. The learning unit stores the model generated by the learning process in the model information storage unit 141. The learning unit also stores the model with updated parameters from the learning process in the model information storage unit 141.

[0144] The learning unit learns various types of information based on the information acquired by the acquisition unit 151. The learning unit learns (generates) a model. The learning unit learns various types of information, such as the model. The learning unit generates a model through learning. The learning unit learns the model using various machine learning techniques. For example, the learning unit learns the parameters of the model (network). The learning unit learns the model using various machine learning techniques.

[0145] The learning unit generates an object detection model M1. The learning unit learns the network parameters. For example, the learning unit learns the network parameters of the object detection model M1.

[0146] The learning unit performs learning processing based on the training data (training data) stored in the memory unit 14. The learning unit generates an object detection model M1 by performing learning processing using the training data stored in the memory unit 14. For example, the learning unit generates a model used for image recognition. The learning unit generates an object detection model M1 by learning the network parameters of the object detection model M1.

[0147] For example, the learning unit performs learning processing using methods such as backpropagation so that the object detection results output by the object detection model M1 approach the ground truth information (Ground-truth in Figure 8, etc.) associated with the image (first image) input to the object detection model M1. For example, the learning unit adjusts the values ​​of the weights (i.e., connection coefficients) that are considered when values ​​are transmitted between nodes during the learning process. In this way, the learning unit learns the object detection model M1 by processing such as backpropagation to correct the parameters (connection coefficients) so that the error between the output of the object detection model M1 and the ground truth information corresponding to the input is reduced. For example, the learning unit generates the object detection model M1 by processing such as backpropagation to minimize a predetermined loss function. For example, the learning unit performs learning processing so that the output of the object detection model M1 (object detection result in Figure 8) to which an image (first image) is input approaches the ground truth information (Ground-truth in Figure 8) associated with the image (first image) input to the object detection model M1. This allows the learning unit to perform a learning process to learn the parameters of the object detection model M1.

[0148] The learning method (learning process) is not limited to the methods described above, and any known technology can be applied. Furthermore, the object detection model M1 may be trained using various conventional machine learning techniques as appropriate. For example, the object detection model M1 may be trained using machine learning techniques such as linear regression or nonlinear regression. For example, the object detection model M1 may be trained using supervised learning machine learning techniques such as SVM (Support Vector Machine). For example, the object detection model M1 may be trained using deep learning techniques. Also, each component of the object detection model M1 (e.g., adaptive processing model M12) may be trained individually.

[0149] Furthermore, the processes related to the embodiments described above may be carried out in various other forms (modifications) besides those described above. For example, the system configuration is not limited to the examples described above and may be in various forms.

[0150] For example, an information processing system may include a device that performs learning processing (learning device) and a device that performs inference processing using the model learned by the learning device (e.g., an information processing device 100). In this case, the information processing system may include both the learning device and the information processing device 100. The above is just one example, and the information processing system may be implemented in various configurations. For example, the information processing device 100 may be a device that performs both learning processing and inference processing. That is, in an information processing system, the learning device and the inference device may be integrated.

[0151] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0152] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0153] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0154] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur.

[0155] <3. Effects of the Disclosure> As described above, the information processing device according to the Disclosure (information processing device 100 in the embodiment) comprises an acquisition unit (acquisition unit 151 in the embodiment), a correction processing unit (correction processing unit 154 in the embodiment), and a generation unit (generation unit 155 in the embodiment). The acquisition unit acquires a first image that is the subject of the task. The correction processing unit uses the feature information of the first image to execute a plurality of correction processes, each of which is a different correction process, in parallel. The generation unit uses a plurality of second images obtained by each of the plurality of correction processes to generate a third image that is the subject of processing related to the task.

[0156] As a result, the information processing device according to this disclosure can generate a third image in which multiple correction processes are reflected in parallel using multiple second images obtained by each of the multiple correction processes, and can appropriately generate an image in which multiple correction processes are reflected.

[0157] Furthermore, the correction processing unit performs multiple correction processes, including gamma correction. The generation unit generates a third image using multiple second images, including a second image on which gamma correction has been performed. As a result, the information processing device can generate a third image in which multiple correction processes, including gamma correction, are reflected in parallel, and can appropriately generate an image that reflects multiple correction processes.

[0158] Furthermore, the correction processing unit performs multiple correction processes, including color correction. The generation unit generates a third image using multiple second images, including a second image on which color correction has been performed. As a result, the information processing device can generate a third image in which multiple correction processes, including color correction, are reflected in parallel, and can appropriately generate an image that reflects multiple correction processes.

[0159] Furthermore, the correction processing unit performs multiple correction processes, including white balance correction. The generation unit generates a third image using multiple second images, including a second image on which white balance correction has been performed. As a result, the information processing device can generate a third image in which multiple correction processes, including white balance correction, are reflected in parallel, and can appropriately generate an image that reflects multiple correction processes.

[0160] Furthermore, the correction processing unit performs multiple correction processes, including brightness correction. The generation unit generates a third image using multiple second images, including a second image on which brightness correction has been performed. As a result, the information processing device can generate a third image in which multiple correction processes, including brightness correction, are reflected in parallel, and can appropriately generate an image that reflects multiple correction processes.

[0161] Furthermore, the generation unit generates a third image by fusing multiple second images. As a result, the information processing device can generate a third image in which multiple correction processes are reflected in parallel, and can appropriately generate an image that reflects multiple correction processes.

[0162] Furthermore, the generation unit generates a third image based on parameters calculated from multiple second images by the CNN. This allows the information processing device to generate a third image in which multiple correction processes are reflected in parallel, and to appropriately generate an image that reflects multiple correction processes.

[0163] Furthermore, the generation unit generates a third image based on the set parameters. This allows the information processing device to generate a third image in which multiple correction processes are reflected in parallel, and to appropriately generate an image that reflects multiple correction processes.

[0164] Furthermore, the generation unit generates a third image by calculating the average of multiple second images. As a result, the information processing device can generate a third image in which multiple correction processes are reflected in parallel, and can appropriately generate an image that reflects multiple correction processes.

[0165] Furthermore, the information processing device includes a feature extraction unit (a feature extraction unit 153 in this embodiment). The feature extraction unit extracts feature information from the first image. The correction processing unit uses the feature information extracted from the first image by the feature extraction unit to perform multiple correction processes. As a result, the information processing device can generate a third image using appropriately extracted feature information, and can appropriately generate an image that reflects the multiple correction processes.

[0166] Furthermore, the information processing device includes a format conversion unit (format conversion unit 152 in this embodiment). The format conversion unit generates an RGB image from a first image, which is a RAW image. The feature extraction unit extracts feature information from the first image using the RGB image. As a result, the information processing device can generate a third image using an appropriately format-converted image, and can appropriately generate an image that reflects multiple correction processes.

[0167] Furthermore, the format conversion unit generates an RGB image based on two G component images, one R component image, and one B component image obtained from the first image, which is a Bayer image. As a result, the information processing device can appropriately generate an RGB image from the Bayer image and appropriately generate an image that reflects multiple correction processes.

[0168] Furthermore, the format conversion unit generates a first RGB image from the first image, which is a Bayer image, containing two G component images, one R component image, and one B component image. From the first RGB image, the two G component images are combined into one G component image, and a second RGB image is generated containing the combined G component image, one R component image, and one B component image. The feature extraction unit extracts feature information from the first image using the second RGB image. As a result, the information processing device can extract feature information using an RGB image appropriately generated from a Bayer image, and can appropriately generate an image that reflects multiple correction processes.

[0169] Furthermore, the format conversion unit generates a first RGB image from the first image, which is a Bayer image, which includes two G component images, one R component image, and one B component image, each half the size of the first image. It then generates a single G component image by calculating the average value of each pixel in the two G component images. As a result, the information processing device can appropriately generate an RGB image containing three component images from the Bayer image, and can appropriately generate an image that reflects multiple correction processes.

[0170] Furthermore, the feature extraction unit extracts feature information from the first image based on CNN. As a result, the information processing device can generate a third image using the appropriately extracted feature information, and can appropriately generate an image that reflects multiple correction processes.

[0171] Furthermore, the correction processing unit uses feature information to estimate the parameters to be used for each of the multiple correction processes, and generates multiple second images by executing each correction process using the estimated parameters corresponding to each correction process. As a result, the information processing device can execute each correction process using the parameters corresponding to each correction process, and can appropriately generate an image that reflects multiple correction processes.

[0172] Furthermore, the correction processing unit generates multiple second images by executing each correction process using parameters and correction functions corresponding to each correction process. This allows the information processing unit to execute each correction process using the corresponding correction function, thereby appropriately generating images that reflect multiple correction processes.

[0173] Furthermore, the correction processing unit generates multiple second images, each of which is a multiple RGB image on which a multiple correction process has been performed. As a result, the information processing device can generate a third image using the multiple RGB images on which each of the multiple correction processes has been performed, thereby appropriately generating an image that reflects the multiple correction processes.

[0174] <4. Hardware Configuration> The information processing device 100 and the like according to the embodiments described above are realized by a computer 1000 having a configuration such as that shown in Figure 9. The information processing device 100 will be used as an example for explanation. Figure 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 has a processing circuitry 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0175] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0176] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0177] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that stores programs for each process of the information processing device 100 according to the embodiment, which is an example of program data 1450.

[0178] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. The communication interface 1500 corresponds to the communication unit 11 of the information processing device 100. For example, the processing circuit 1100 receives data from other devices or transmits data generated by the processing circuit 1100 to other devices via the communication interface 1500.

[0179] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0180] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (Organic Electro Luminescence Display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device.

[0181] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0182] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the processing circuit 1100 of the computer 1000 functions as a control unit 15 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the information processing program according to this disclosure and various data stored by the storage unit 14. The processing circuit 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside the computer 1000, but may be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0183] Furthermore, this technology can also take the following configurations: (1) An information processing device comprising: an acquisition unit that acquires a first image that is the target of a task; a correction processing unit that executes a plurality of correction processes, each being a different correction process, in parallel using the characteristic information of the first image; and a generation unit that generates a third image to be used for processing related to the task using a plurality of second images obtained by each of the plurality of correction processes. (2) The information processing device according to (1), wherein the correction processing unit executes the plurality of correction processes, including gamma correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the gamma correction has been performed. (3) The information processing device according to (1) or (2), wherein the correction processing unit executes the plurality of correction processes, including color correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the color correction has been performed. (4) The correction processing unit performs the plurality of correction processes, including white balance correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the white balance correction has been performed. The information processing device according to any one of (1) to (3). (5) The correction processing unit performs the plurality of correction processes, including brightness correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the brightness correction has been performed. The information processing device according to any one of (1) to (4). (6) The generation unit generates the third image by a process of fusing the plurality of second images. The information processing device according to any one of (1) to (5). (7) The generation unit generates the third image based on parameters calculated from the plurality of second images by a CNN (Convolutional Neural Network). The information processing device according to (6). (8) The generation unit generates the third image based on set parameters. The information processing device according to (6). (9) The information processing apparatus according to (6), wherein the generation unit generates the third image by averaging the plurality of second images for each corresponding pixel.(10) An information processing device according to any one of (1) to (9), comprising: a feature extraction unit for extracting the feature information of the first image, wherein the correction processing unit performs the plurality of correction processes using the feature information extracted from the first image by the feature extraction unit. (11) An information processing device according to (10), comprising: a format conversion unit for generating an RGB (Red Green Blue) image from the first image which is a RAW (Raw Image Format) image, wherein the feature extraction unit extracts the feature information of the first image using the RGB image. (12) An information processing device according to (11), wherein the format conversion unit generates the RGB image based on two G component images, one R component image, and one B component image obtained from the first image which is a Bayer image. (13) The format conversion unit generates a first RGB image from the first image, which is a Bayer image, which includes two G component images, one R component image, and one B component image; from the first RGB image, which includes the two G component images, which include one G component image, which includes one B component image; and the feature extraction unit extracts the feature information of the first image using the second RGB image, as described in (11) or (12). (14) The format conversion unit generates a first RGB image from the first image, which is a Bayer image, which includes two G component images, one R component image, and one B component image that are half the size of the first image; and generates a single G component image by calculating the average value of each pixel of the two G component images, as described in (13). (15) The feature extraction unit extracts the feature information of the first image based on CNN. The information processing device according to any one of (10) to (14). (16) The correction processing unit estimates the parameters to be used for each of the plurality of correction processes using the feature information, and generates the plurality of second images by executing each of the correction processes using the estimated parameters corresponding to each correction process. The information processing device according to any one of (1) to (15).(17) The information processing apparatus according to (16), wherein the correction processing unit generates the plurality of second images by executing each of the correction processes using the parameters corresponding to each of the correction processes and the correction function corresponding to each of the correction processes. (18) The information processing apparatus according to (16) or (17), wherein the correction processing unit generates the plurality of second images which are a plurality of RGB images in which each of the plurality of correction processes has been executed. (19) An information processing method comprising: acquiring a first image which is the subject of a task; executing a plurality of correction processes which are each different correction processes in parallel using the feature information of the first image; and generating a third image which is the subject of processing related to the task using a plurality of second images obtained by each of the plurality of correction processes. (20) An information processing program that causes a computer to perform the following: acquire a first image which is the subject of the task; execute a plurality of correction processes, each being a different correction process, in parallel using the feature information of the first image; and generate a third image which is the subject of the task which is to be used for processing related to the task, using a plurality of second images obtained by each of the plurality of correction processes.

[0184] 100 Information Processing Device 11 Communication Unit 12 Input Unit 13 Output Unit 14 Storage Unit 141 Model Information Storage Unit 15 Control Unit 151 Acquisition Unit 152 Format Conversion Unit 153 Feature Extraction Unit 154 Correction Processing Unit 155 Generation Unit 156 Task Processing Unit 157 Transmission Unit

Claims

1. An information processing device comprising: an acquisition unit that acquires a first image that is the target of a task; a correction processing unit that executes a plurality of correction processes, each being a different correction process, in parallel using the characteristic information of the first image; and a generation unit that generates a third image to be used for processing related to the task using a plurality of second images obtained by each of the plurality of correction processes.

2. The information processing apparatus according to claim 1, wherein the correction processing unit performs the plurality of correction processes, including gamma correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the gamma correction has been performed.

3. The information processing apparatus according to claim 1, wherein the correction processing unit performs the plurality of correction processes, including color correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the color correction has been performed.

4. The information processing apparatus according to claim 1, wherein the correction processing unit performs the plurality of correction processes, including white balance correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the white balance correction has been performed.

5. The information processing apparatus according to claim 1, wherein the correction processing unit performs the plurality of correction processes, including brightness correction, and the generation unit generates the third image using the plurality of second images, including the second image on which the brightness correction has been performed.

6. The information processing apparatus according to claim 1, wherein the generation unit generates the third image by a process of fusing the plurality of second images.

7. The information processing apparatus according to claim 6, wherein the generation unit generates the third image based on parameters calculated from the plurality of second images by a CNN (Convolutional Neural Network).

8. The information processing apparatus according to claim 6, wherein the generation unit generates the third image based on the set parameters.

9. The information processing apparatus according to claim 6, wherein the generation unit generates the third image by averaging the plurality of second images for each corresponding pixel.

10. The information processing apparatus according to claim 1, comprising: a feature extraction unit for extracting the feature information of the first image, wherein the correction processing unit performs the plurality of correction processes using the feature information extracted from the first image by the feature extraction unit.

11. The information processing apparatus according to claim 10, comprising: a format conversion unit that generates an RGB (Red Green Blue) image from the first image which is a RAW (Raw Image Format) image, wherein the feature extraction unit extracts the feature information of the first image using the RGB image.

12. The information processing apparatus according to claim 11, wherein the format conversion unit generates the RGB image based on two G component images, one R component image, and one B component image obtained from the first image which is a Bayer image.

13. The information processing apparatus according to claim 11, wherein the format conversion unit generates a first RGB image from the first image which is a Bayer image, which includes two G component images, one R component image, and one B component image; the two G component images from the first RGB image are combined into one G component image to generate a second RGB image which includes the combined one G component image, the one R component image, and the one B component image; and the feature extraction unit extracts the feature information of the first image using the second RGB image.

14. The information processing apparatus according to claim 13, wherein the format conversion unit generates a first RGB image from the first image, which is a Bayer image, which includes two G component images having half the size of the first image, one R component image, and one B component image, and generates the one G component image by calculating the average value of each pixel of the two G component images.

15. The information processing apparatus according to claim 10, wherein the feature extraction unit extracts the feature information of the first image based on CNN.

16. The information processing apparatus according to claim 1, wherein the correction processing unit estimates parameters to be used for each of the plurality of correction processes using the feature information, and generates the plurality of second images by executing each of the correction processes using the estimated parameters corresponding to each correction process.

17. The information processing apparatus according to claim 16, wherein the correction processing unit generates the plurality of second images by executing each correction process using the parameters corresponding to each correction process and the correction function corresponding to each correction process.

18. The information processing apparatus according to claim 16, wherein the correction processing unit generates a plurality of second images which are a plurality of RGB images on which each of the plurality of correction processes has been performed.

19. An information processing method comprising: acquiring a first image that is the subject of a task; executing a plurality of correction processes, each being a different correction process, in parallel using the feature information of the first image; and generating a third image to be used for processing related to the task using a plurality of second images obtained by each of the plurality of correction processes.

20. An information processing program that causes a computer to perform the following: acquire a first image that is the target of the task; execute multiple correction processes, each being a different correction process, in parallel using the feature information of the first image; and generate a third image to be used for processing related to the task using multiple second images obtained by each of the multiple correction processes.

Images