Image processing method, image processing device, imaging device, image processing system, and program
By recording editing information in the metadata of image files, the method addresses the lack of reproducibility in image processing, ensuring consistent and efficient editing results across different environments and times.
Patent Information
- Application Number
- PCT/JP2026/015512
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2026-04-17
- Publication Date
- 2026-11-12
AI Technical Summary
Existing image processing methods using generation models lack reproducibility, as they require both an image and a prompt for consistent editing results, and recreating prompts is time-consuming and laborious.
An image processing method that records editing information, such as prompts, in the metadata of the image file to ensure reproducibility by associating it with the image, allowing for consistent and efficient editing across different environments and times.
Ensures high reproducibility of image editing by managing editing information with the image, reducing the need for manual prompt recreation and maintaining image quality, thus improving efficiency and reliability.
Smart Images

Figure JP2026015512_12112026_PF_FP_ABST
Abstract
Description
Image processing method, image processing apparatus, imaging apparatus, image processing system, and program
[0001] The present disclosure relates to an image processing method, an image processing apparatus, an imaging apparatus, an image processing system, and a program.
[0002] Non-Patent Document 1 discloses a method of executing image editing in consideration of the meaning of text described in a prompt by inputting the prompt and an image into a generation model. In this method, a large amount of learning data is generated using GPT-3 (registered trademark) and Stable Diffusion, and a generation model is trained using the learning data, thereby enabling editing such as object replacement and background change.
[0003] "InstructPix2Pix: Learning to Follow Image Editing Instructions", Tim Brooks, Alexander Holynski, Alexei A. Efros
[0004] In image processing using a generation model, improvement of reproducibility is demanded.
[0005] An image processing method as one aspect of the present disclosure is characterized by comprising a step of acquiring first information including a prompt for a first image, a step of generating a second image based on the first image and the first information using a generation model, and a step of associating the first information with a third image based on the first image.
[0006] According to the present disclosure, an image processing method with high reproducibility using a generation model can be provided.
[0007] This is a flowchart of the corrected image generation in Example 1. This is a block diagram of the imaging device in Example 1. This is a schematic diagram of the input image in Example 1. This is a schematic diagram of the editing information in Example 1. This is an explanatory diagram of the diffusion model in Example 1. This is a schematic diagram of the edited image in Example 1. This is a diagram showing an example of a function used for gamma correction in Example 1. This is an explanatory diagram of the demosaiking process in Example 1. This is a diagram of the filter used for linear interpolation in Example 1. This is an explanatory diagram of the demosaiking method for R in Example 1. This is a flowchart of the regeneration process in Example 1. This is a block diagram of the image processing system in Example 2. This is an external view of the image processing system in Example 2. This is a flowchart of the edited image generation in Example 2. This is a schematic diagram of the input image in Example 2. This is a diagram showing an example of the editing region map in Example 2. This is a schematic diagram of the edited image in Example 2.
[0008] The embodiments of this disclosure will be described in detail below with reference to the drawings. In each drawing, the same reference numerals are used for the same components, and redundant descriptions are omitted.
[0009] First, before describing each embodiment in detail, I will explain the gist of this disclosure. This disclosure is a technology relating to the recording of editing information such as prompts used in image generation. When editing an image by providing editing information to the image, it is not possible to output the same image unless the same information is provided. When generating an image from a prompt, it is sufficient to record only the prompt, but when generating an image using a combination of an image and a prompt, both pieces of information are necessary. If an image used for generation exists but a prompt does not exist, a prompt must be created again. In this case, if the prompt used is different from the one used when the process was first executed, it is not possible to generate the same edited image. Therefore, in order to ensure reproducibility, it is desirable to manage the image before editing and the editing information used for editing together. Based on this idea, this disclosure records the editing information used for editing in the metadata portion of the image before editing. This makes it easy to re-execute the editing process and ensures the reproducibility of the process.
[0010] First, the imaging device 100 in Embodiment 1 will be described. Figure 2 is a block diagram of the imaging device 100 in this embodiment. The imaging device 100 includes an optical system 101, an image sensor 102, an image processing unit (image processing device) 103, a recording medium 104, a display unit 105, and a system controller 106.
[0011] The optical system 101 collects light incident from the subject space to the imaging device 100. The image sensor 102 receives (photoelectrically converts) the optical image (subject image) formed through the optical system 101 to acquire a captured image. The image sensor 102 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The captured image acquired by the imaging device 100 includes blurring due to aberrations and diffraction of the optical system 101 and noise from the image sensor 102.
[0012] The image processing unit 103 is a device that performs correction processing on captured images and includes an acquisition unit 103a, a generation unit 103b, and a recording unit 103c. The acquisition unit 103a acquires captured images and input information from the user. The acquired captured images are recorded on the recording medium 104 by the recording unit 103c. The generation unit 103b performs calculations for the generation model, etc. The generated image (edited image) output by the generation model is recorded on the recording medium 104 by the recording unit 103c. In this embodiment, the imaging device 100 is a digital camera, but it is not limited to this and may be an electronic device such as a smartphone with a camera or a personal computer.
[0013] Furthermore, captured images and generated images are output to at least one of the display unit 105 and the recording medium 104. The display unit 105 is, for example, a liquid crystal display or a projector. Note that it does not have to be a separate liquid crystal display, but may be a liquid crystal display within the imaging device 100. The user can check the image in progress via the display unit 105 and perform image editing work, etc., via an input device (not shown). The recording medium 104 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device is, for example, a keyboard or a mouse.
[0014] The above series of controls are performed by the system controller 106.
[0015] Next, the generation of edited images (generated images) using the generation model executed by the image processing unit 103 in this embodiment will be described. Figure 1 is a flowchart relating to the generation of edited images. Each step in Figure 1 is mainly executed by the acquisition unit 103a, the generation unit 103b, or the recording unit 103c of the image processing unit 103.
[0016] First, in step S101, the acquisition unit 103a acquires the input image (first image). In this embodiment, a RAW image sent from the image sensor 102 is used as the input image. Here, a RAW image is undeveloped captured image data output from the image sensor 102, in which the light intensity and brightness value of each pixel have an approximately linear relationship. The RAW image is developed before the user views the image, and gamma correction is performed at that time. In other words, in this embodiment, the input image is the image before gamma correction. Gamma correction is a process that raises the input brightness value to a power, for example, 1 / 2.2 is used as the exponent.
[0017] Figure 3 is a schematic diagram of the input image acquired in step S101. Figure 3 is modeled after a snapshot taken in an art museum. Two people are pictured in the foreground as the main subjects. In the background, one person can be seen who was unintentionally included in the image. In this embodiment, we will describe the process of generating an image from which the background person has been removed by editing using a generative model.
[0018] Next, in step S102, the acquisition unit 103a acquires editing information (first information). Figure 4 is a schematic diagram showing the editing information. In this embodiment, the editing information includes a text-formatted prompt (a prompt for the input image). Both voice input and text input are possible for setting the prompt. User input via the user interface becomes the editing information, and image editing is performed based on this. In the case of voice input, the user verbally instructs the editing content using a microphone. Voice recognition technology converts the voice into text and acquires it as editing information. In the case of text input, the user inputs the editing content using a keyboard or touchscreen. These input methods allow the user to provide editing information intuitively and flexibly.
[0019] In this embodiment, the editing information is a prompt, but it is not limited to this. A two-dimensional mask image or depth information may be used as the editing information. For example, a mask image can be used to select a specific area and edit only that part. Using depth information allows for editing based on the depth information of the image. Negative prompts can also be used as editing information. Negative prompts are used to specify elements to be removed from the image or effects that should not be applied. For example, instructions such as "blurred" or "low quality." By using negative prompts, the specified elements are excluded, enabling more precise and intentional image editing.
[0020] In this embodiment, the editing information may also include information about a seed number (a number that changes the editing result depending on the number) in addition to the prompt.
[0021] Next, in step S103, the generation unit 103b generates an edited image (second image) (performs editing processing) based on the input image acquired in step S101 and the editing information acquired in step S102 (by inputting the editing information into the generation model). The generation model is a model that acquires a probability distribution for generating various desired data through learning and generates data according to that probability distribution. The edited image is generated using the generation model.
[0022] Generative models include GANs (Generative Adversarial Networks) and VAEs (Variational Autoencoders). Other generative models include diffusion models and flow-based generative models. In this example, a diffusion model is used as the first machine learning model. However, other generative models may also be used. First, the characteristics of other generative models will be described, and finally, the diffusion model used in this example will be explained in detail.
[0023] GANs learn by having two neural networks, a generative model and a discriminative model, compete against each other. The generative model generates data that resembles real data, while the discriminative model determines whether that data is real or fake. Through this competition, the generative model learns to generate more realistic data.
[0024] A VAE (Value-Added Encoding) encodes data into a latent variable space and generates new data by decoding the data from those latent variables. VAEs take a probabilistic approach and learn the distribution of latent variables. As a result, the generated data is diverse and allows for continuous transformation.
[0025] Flow-based generative models learn reversible transformations between data and latent variables. These models model the data generation process by maximizing the log-likelihood. A series of processes or operations performed by a generative model to achieve a specific objective is called a task. For example, in a style transfer task, the model transforms the style of an input image to a painterly style, for instance.
[0026] Referring to Figure 5, an overview of the diffusion model for image generation will be explained. Figure 5 is an explanatory diagram of the diffusion model. In the diffusion model, noise is added stepwise to an image with desired properties, and a diffusion process is considered in which noise is added stepwise until a completely noisy image is obtained. Conversely, noise is removed stepwise from a noisy image to obtain an image with desired properties. Here, the noise is Gaussian noise. Each step is represented by time t, where t=0 is the state of the image with desired properties, and t=T is the state of the completely noisy image. Note that the diffusion model is inspired by thermodynamics, so the expression "time" is used for convenience, but time does not actually change, and t can also be expressed as a calculation step or noise intensity level.
[0027] The diffusion process is easy to perform because it adds noise to the image, but in the dediffusion process, even if you subtract appropriate Gaussian noise, you cannot obtain an image with the desired properties. Therefore, in the diffusion model, a neural network is used for noise reduction at each time step of the dediffusion process. By repeatedly performing noise reduction using a neural network on the complete noise image at t=T, we obtain the noise image at t=0 (an image with the desired properties).
[0028] Furthermore, in the diffusion model, the output can be controlled by inputting constraint images and text-based prompts in addition to the noise image. Figure 6 is a schematic diagram of the edited image. Compared to the input image before editing shown in Figure 3, it can be seen that the person located in the background has been removed. In this way, by using the generative model, it is possible to generate an edited image that takes into account the prompt information shown in Figure 4.
[0029] In this embodiment, we have described how to remove a person, but editing is not limited to this; various editing processes can be performed by changing the prompts given. For example, if you want to change the main subject, you can enter a prompt related to the main subject. For example, by setting prompts such as "Make the person in this photo smile" or "Change the hairstyle of the person on the right to long hair," you can make changes to the main subject.
[0030] Furthermore, changes can be made to the background. For example, by prompting the model to "change the background to a seaside landscape" or "replace the background with a city night view," the model can change the background to match the prompt from the museum. By using a diffusion model, it is possible to generate an image that blends well with the foreground figure even when the background is changed. Specifically, when changing the background, it is desirable to set prompts that take into account lighting conditions, color tones, and textures suitable for the foreground figure. For example, by giving the instruction "change the background to a sunset that harmonizes with this person," the generation model will generate an image while maintaining a sense of unity between the foreground figure and the new background. In this way, the background and foreground are integrated, resulting in a visually natural finish.
[0031] Next, in step S104, the generation unit 103b performs development processing on the edited image generated in step S103. In this embodiment, the development processing includes gamma correction, white balance correction, and demosaicing.
[0032] First, let's discuss gamma correction. A RAW image is undeveloped image data output from the image sensor 102, where the light intensity and brightness value of each pixel are in a roughly linear relationship. Normally, before the user views the image, development processing and various correction processes are performed on the RAW image. At that time, brightness-related corrections such as exposure correction and peripheral illumination correction, as well as gamma correction and gradation conversion, are applied. Brightness-related correction processes such as exposure correction and peripheral illumination correction can make problems in low-brightness or high-brightness areas more noticeable. Gamma correction is a process that raises the input brightness value to a power, and a power exponent such as 1 / 2.2 is used.
[0033] Figure 7 shows an example of a function used for gamma correction. In Figure 7, the horizontal axis represents the luminance value before gamma correction, and the vertical axis represents the luminance value after gamma correction. As shown in Figure 7, gamma correction is a process in which the curve showing the relationship between luminance values before and after correction (gamma curve) is located at or above a straight line with a slope of 1 (the dashed line in Figure 7). Because such a conversion process is performed during development, the signal values on the low-luminance side are stretched significantly compared to the high-luminance side. In other words, the drawbacks on the low-luminance side in the RAW image become more noticeable after gamma correction.
[0034] Next, we will discuss white balance correction. White balance correction is a process that adjusts the brightness levels by applying gain processing to each color component of the RGB components output by the image sensor 102, so that the achromatic parts of the subject appear achromatic in the output image. Without white balance correction, the color characteristics of the image sensor 102 will not correctly reproduce the color of the subject, resulting in an image with colors different from the actual subject. Normally, there is a white balance coefficient for each RGB color component, so white balance correction can be performed by multiplying these coefficients by the corresponding color components of the captured image. If the black level is offset, the offset is subtracted before performing white balance correction. Also, if the white balance coefficients are separated for G1 and G2, their average may be used as the white balance coefficient for G.
[0035] Next, the demosaiking process will be explained with reference to Figures 8(a) to 8(c). Figures 8(a) to 8(c) are explanatory diagrams regarding the color components of the input image. In this embodiment, the input image is a Bayer array image as shown in Figure 8(a). Here, RGB represents red, green, and blue, respectively. Since there are two types of G in the Bayer array, G1 and G2, the demosaiking process differs for G and RB. Figure 8(b) shows the state after extracting only G1 and G2 from Figure 8(a). When performing demosaiking on the G in Figure 8(a), after extracting G as shown in Figure 8(b), linear interpolation is performed. Figure 9(a) shows the filter used for linear interpolation. By convolving the filter in Figure 9(a) onto the G in Figure 8(b), the interpolated G shown in Figure 8(c) can be generated.
[0036] Next, we will explain the demosaicing process for RB. Since both R and B are of only one type and the same process can be applied to them, in this embodiment we will only explain the process for R. Figures 10(a) to (c) are explanatory diagrams of the demosaicing method for R. Figure 10(a) shows the state in which only R is extracted from the Bayer array in Figure 8(a). In this embodiment, color difference information is calculated by taking the difference between R and G. Figure 10(b) shows the color difference of R before demosaicing, and the difference between the R values in Figure 10(a) and the G values after demosaicing in Figure 8(c) is taken only for pixels where R values exist.
[0037] Next, interpolation is performed on the color difference in Figure 10(b). Figure 9(b) shows an example of a filter for performing interpolation. By convolving the filter shown in Figure 9(b), color difference information corresponding to all pixels can be calculated. Figure 10(c) shows R after demosaiking, which can be calculated by adding G to the color difference after convolving the filter in Figure 9(b). The demosaiking process for R has been described above, but the same demosaiking process can be performed on B.
[0038] In this embodiment, interpolation is performed using color difference for RB, but interpolation may also be performed directly on RB. Furthermore, in addition to simple interpolation, adaptive processing using the gradient of RGB values may be incorporated to perform demosaicing.
[0039] In this embodiment, gamma correction, white balance correction, and demosaicing were described as development processes, but there are various other processes such as exposure correction and peripheral illumination correction. Although not explained here, these various methods may also be applied in this embodiment. In this embodiment, development processing is performed on the edited image. By generating an edited image from a RAW image in this way, a high-quality edited image can be produced. This is because RAW images usually have a higher tonal range than developed images.
[0040] On the other hand, in developed images, information that was present in the RAW image may be lost during the development process. Thus, images that have lost information due to the development process before being input into the generative model are not taken into account by the generative model, which can lead to unintended editing results. For example, as mentioned above, in gamma correction, the tonal range on the high-luminance side is reduced by development, and the tonal range itself also decreases, making this problem more likely to occur.
[0041] Therefore, in this embodiment, editing is performed on the RAW image, and development is performed on the resulting edited image. Note that this order is not mandatory; development may be performed first, followed by editing. In other words, the input image may be a gamma-corrected image (developed image). Depending on the input image, the effects of development, such as tonal reduction, may be minimal, so development can also be performed first.
[0042] Next, in step S105, the system controller 106 displays the edited image generated (developed) in step S104 on the display unit 105. At this time, the display unit 105 displays, in addition to the edited image, a confirmation message for confirming with the user whether to confirm the edited result, as well as a "Yes" button and a "No" button. For example, "Do you want to confirm the edited result?" is displayed as the confirmation message. When the user presses the "Yes" button, the edited result is confirmed, and when the user presses the "No" button, the editing can be redone. Further, the display unit 105 may perform highlight display so that the edited portion can be recognized. Alternatively, the pre-editing and post-editing images may be displayed side by side, or may be displayed in a switchable manner. This allows the user to intuitively check the edited result and make an appropriate determination.
[0043] Next, in step S106, the system controller 106 determines whether or not to end the editing process. If the "No" button is pressed in step S105, the system controller 106 determines that the editing process has not been completed, and returns to step S102. On the other hand, if the "Yes" button is pressed in step S105, the system controller 106 determines that the editing process has been completed, and proceeds to step S107.
[0044] Next, in step S107, the recording unit 103c records the edited image on the recording medium 104. At this time, as the edited image to be recorded, not only the developed image but also the pre-development edited image may be recorded. Further, the recording destination of the edited image is not limited to the recording medium 104 in the imaging apparatus 100. For example, it is also possible to connect the imaging apparatus 100 to a PC and record the image on a recording medium on the PC side (for example, a hard disk or an SSD). This provides a wider variety of storage methods.
[0045] Next, in step S108, the recording unit 103c records editing information for the input image acquired in step S101. Specifically, it adds editing information to the metadata of the input image. Metadata is data embedded in the image file and includes the image shooting conditions and editing history. That is, in step S108, the recording unit 103c associates the editing information with the input image and records it on the recording medium 104. The metadata may also include information about the edited image. Information about the edited image may include, for example, the file name and file path of the edited image. Alternatively, an identification number may be included in the metadata of both the input image and the edited image so that it can be seen that they correspond to each other.
[0046] In this embodiment, the image to which editing information is associated is not limited to the input image acquired in step S101. The recording unit 103c may associate and record the editing information with an image based on the input image (third image). The image based on the input image is not limited to the input image itself acquired in step S101, but may also be an image that is substantially equivalent to the input image, such as a duplicate of the input image (duplicate image), or an image that has been corrected by applying arbitrary processing to the input image (corrected image). The third image is the image before it is input to the generation model.
[0047] Images captured by cameras generally use the Exif (Exchangeable Image File Format) format. The Exif format is a standard format for embedding metadata such as the date and time of capture, camera settings, and location information into image files. This makes it possible to check detailed information about the image later. In this embodiment, the Exif format is used to record editing information such as prompts used in the generative model, negative prompts, random number seeds used in the generative model, and information about the generative model (sampler, number of steps, version).
[0048] Also, depending on the generation model, prompts preset by the user to output an intended image may be adjusted in advance. In this case, the edit information to be recorded may be either the prompt specified by the user or the adjusted prompt, or both. This makes it possible to improve the quality and consistency of the generated image.
[0049] In this way, by recording edit information in an input image, a consistent edited image can be obtained when the same input image is used. This makes it possible to reproduce similar results even in different environments and at different times. Furthermore, when performing a plurality of editing processes, the edit information may be updated in the meta information of the input image for each single editing process. When the edit information is updated, the updated edit information can be recorded in association with the third image. Such updates make it possible to retain a detailed editing history. Further, the update of the edit information can be performed in the form of appending or overwriting, and the apparatuses do not need to be the same. When the edit information is appended and saved, the edit information may be grouped and saved. For example, by storing information such that group (1) corresponds to edit 1 and edit 2, and group (2) corresponds to edit 3 and edit 4, only required editing processes can be executed by reading out a desired group to be executed at the time of re-execution. Common editing processes may be included in each group. This makes it possible to consistently manage the editing history even across different devices and software environments.
[0050] The above concludes the description regarding generation of an edited image using a generation model. In the present embodiment, a method of processing one captured image as an input image has been described, but the captured image may be divided, and divided images (each of a plurality of partial regions (image regions) divided in the captured image) may be processed as input images. Dividing an image has several merits. For example, processing divided images (partial regions in the image) reduces the computational load and improves processing speed compared to processing the entire image. Furthermore, the division enables detailed editing for a specific region, and highly accurate image generation can be expected.
[0051] In this embodiment, it is also possible to change the prompt depending on the segmented image (for each sub-region). For example, in the case of a prompt like "Please delete the person in the background" as shown in Figure 4, if the target person is not present in the segmented image, it may be processed incorrectly. In such cases, setting an appropriate prompt for each segmented image can enable more accurate editing.
[0052] On the other hand, negative prompts may not change due to splitting. For example, prompts such as "blurry" or "low quality" may have the same effect regardless of whether splitting is performed or not. In such cases, the prompt can be applied without considering the effect of splitting.
[0053] Furthermore, different prompts (first prompt) and other editing information for each segmented image can be recorded as metadata for the input image. If a common prompt (second prompt) is used for each segmented image, as with the negative prompt mentioned above, it can be recorded as common editing information. This is useful when reviewing or re-editing edits later. In addition, if the image is segmented, recording the segment size as editing information enables highly reproducible editing.
[0054] As described above, the method of dividing and processing captured images has many advantages, and is expected to lead to more efficient and accurate image generation. In this embodiment, by using these methods, advanced image editing utilizing the generation model becomes possible.
[0055] Next, the regeneration process of the edited image using the generation model executed by the image processing unit 103 in this embodiment will be described. The regeneration process is a process for re-executing the image editing process using the generation model described with reference to Figure 1. Figure 11 is a flowchart relating to the regeneration of the edited image. Each step in Figure 11 is mainly executed by the acquisition unit 103a, the generation unit 103b, and the recording unit 103c of the image processing unit 103. The regeneration process shown in Figure 11 differs from the editing process shown in Figure 1 in step S202, so this process will be explained. Steps S201, S203, S204, and S205 in Figure 1 correspond to steps S101, S103, S104, and S107 in Figure 1, respectively, so their explanations will be omitted.
[0056] In step S202, the acquisition unit 103a acquires editing information from the metadata of the input image. The editing information acquired includes prompts, negative prompts, the seed of random numbers used in the generative model, or information related to the generative model (sampler, number of steps, version), etc., recorded in step S108. Furthermore, if the image region is divided and editing is performed, the acquisition unit 103a also acquires the division information as editing information. This makes it possible to use all editing information, including the division information, when reproducing previous results, thus enabling more accurate reproduction.
[0057] The processing from step S203 onward is omitted as it overlaps with the explanation in Figure 1. However, when recording the edited image, it is possible to not only save it to the recording medium 104 but also display the editing result on the display unit 105. It is also possible to check the displayed editing result and perform the editing work again. When editing again, the editing information used for the re-editing can be recorded in the metadata of the input image. This makes it possible to manage the history of re-editing in detail.
[0058] Furthermore, depending on the image file format, it is possible to save multiple images. For example, TIFF (Tagged Image File Format) is a format that allows multiple images to be saved in a single file. The TIFF format is a format for saving high-quality images and has the characteristic of being able to save multiple images in one file. This makes it possible to save images before and after editing, or different edited versions, together in one file. In addition, since the TIFF format can be saved in an uncompressed format, it is possible to save images without compromising image quality. In this embodiment, we will describe the case of the TIFF format, but other formats may also be used.
[0059] Furthermore, in addition to editing information, the edited image (second image) itself may be added to the input image. In other words, the second image may be recorded in association with the third image. By saving the edited image together in this way, it becomes possible to check the edited content later. For example, by comparing the original image with the edited image, it is possible to visually confirm what changes have been made.
[0060] As described above, this embodiment employs a method of dividing an image region, performing editing processing, and acquiring that information as editing information. Furthermore, by using the editing information recorded in the input image, it is possible to reproduce previous results. In addition, when recording the edited image, it is possible not only to save it on a recording medium but also to display the editing result on the display unit and perform the editing work again. When re-editing, the editing information used for re-editing can be recorded in the metadata of the input image, and the history of re-editing can be managed in detail.
[0061] This allows for detailed management of the image generation process and editing history, significantly improving reliability and efficiency. In particular, using the TIFF format allows for saving multiple images in high quality, and enables saving pre- and post-edit images, or different edited versions, in a single file. This allows for centralized management of the image editing history, which is extremely convenient when reviewing or re-editing edits later.
[0062] The above is a detailed explanation of the generation of edited images and the recording of editing information in this embodiment. By using these methods, more efficient and accurate image editing becomes possible.
[0063] In this embodiment, it is important to record editing information (such as prompts) on the input image. Recording editing information on the input image makes data management easier, and the information does not become scattered, allowing for centralized management. This makes it easier to track the editing history, which is very convenient when re-editing or regenerating.
[0064] One of the advantages of recording editing information with the input image is ensuring reproducibility. If editing information is recorded, the same result can be reproduced by re-running the process using the same prompts and settings. If editing information is not recorded, the prompts must be recreated, which is an unnecessary task. Furthermore, recreating prompts is difficult to automate and must be done manually, which is time-consuming and laborious. Even if they can be recreated, if the prompts do not match, the same generated image cannot be produced. The same applies to the seed value. Even if the prompts match, if the seed value is different, the generated image will also be different. Therefore, it is important to accurately record both the prompts and the seed value.
[0065] On the other hand, it is also possible to record editing information in the metadata of the edited image rather than the input image. However, re-editing using the edited image and editing information may result in significant image quality degradation. In particular, if the input image is a RAW image and the edited image is a developed image, the degradation in image quality due to re-editing is noticeable. Therefore, it is preferable to save the editing information in the input image.
[0066] As described above, recording editing information in the input image simplifies data management, ensures reproducibility, and improves the efficiency of re-editing. Furthermore, accurately recording editing information allows for regeneration under the same conditions, enabling highly reliable image editing.
[0067] Next, with reference to Figure 12, the image processing system 200 in Embodiment 2 will be described. In this embodiment, the generation of the edited image is performed by the generation unit 202b in the image processing device 202. Also, the image generation flow in this embodiment differs from that of Embodiment 1.
[0068] Figure 12 is a block diagram of the image processing system 200 in this embodiment. Figure 13 is an external view of the image processing system 200. The image processing system 200 includes an imaging device 201, an image processing device 202, a display device 203, a recording medium 204, and an output device 205.
[0069] The imaging device 201 includes an optical system 201a and an image sensor 201b. The optical system 201a collects light incident on the imaging device 201 from the subject space. The image sensor 201b receives (photoelectrically converts) the optical image (subject image) formed through the optical system 201a and acquires a captured image.
[0070] The image processing device 202 is a device that performs correction processing on captured images and has an acquisition unit 202a, a generation unit 202b, and a recording unit 202c. The acquisition unit 202a acquires captured images and input information from the user. The acquired captured images are recorded on the recording medium 204 by the recording unit 202c. The generation unit 202b performs calculations of the generation model, etc. The generated image output by the generation model is recorded on the recording medium 204 by the recording unit 202c. In this embodiment, the image processing device 202 is a personal computer (PC), but is not limited to this and may be other electronic devices such as smartphones or tablets.
[0071] Furthermore, the captured image and the generated image are output to at least one of the display device 203 and the recording medium 204. The display device 203 is, for example, a liquid crystal display or a projector. Note that the display device 203 may be a liquid crystal display within the image processing device 202 instead of a separate liquid crystal display. The user can check the image in progress via the display device 203 and perform image editing work, etc., via an input device (not shown). The recording medium 204 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device is, for example, a keyboard or a mouse.
[0072] Next, with reference to Figure 14, the generation of edited images performed by the image processing device 202 in this embodiment will be described. Figure 14 is a flowchart relating to the generation of edited images. Each step in Figure 14 is mainly performed by the acquisition unit 202a, the generation unit 202b, or the recording unit 202c of the image processing device 202. Steps S301 and S302 are the same as steps S101 and S102 performed by the image processing unit 103 in Embodiment 1, respectively.
[0073] In step S303, the generation unit 202b creates a map to be used for editing the input image. In Example 1, a prompt is used as editing information, but in this embodiment, both the prompt and the map are used as editing information. Figure 15 is a schematic diagram of the input image acquired in step S301. In Example 1, an image before development was used as the input image, but in this embodiment, an image after development is used as the input image. Also, the editing information acquired in step S302 is assumed to be a text prompt that says, "Please change the background." To create the map, the input image is displayed on the display device 203, and the user looks at the displayed input image and selects the area they want to edit. Figure 16 is an example of a map, showing that the background area is selected by the shaded area. By specifying the area to be edited in this way, the result intended by the user is reflected more accurately.
[0074] While we've described a method where the user selects the map, it's also possible to create maps automatically. For example, one method involves estimating the distance to each subject from an image and generating a depth map. A depth map contains distance information for each subject in an image, making it easier to distinguish between the background and foreground. Furthermore, maps are not limited to depth information or user-selected areas; pose information of a person can also be used as part of the map. By utilizing pose information, editing based on the person's movement and posture becomes possible, resulting in more natural and realistic editing results.
[0075] This embodiment allows users to intuitively specify the area they want to edit, thus streamlining the editing process. By directly selecting the editing area, users can more easily achieve their intended editing results. Furthermore, automatically generating maps reduces user effort and enables rapid editing. For example, using a depth map allows for automatic identification of the background and foreground, making it easy to change the background or emphasize the foreground. Moreover, utilizing various maps, such as pose information, enables more advanced and precise editing. This broadens the scope of image editing and allows for meeting diverse needs.
[0076] Next, in step S304, the generation unit 202b performs editing of the input image using the input image acquired in step S301, the prompt acquired in step S302, and the map information (mask) generated in step S303. In Example 1, one image was generated, but in this example, multiple edited images are generated.
[0077] Figures 17(a) and 17(b) are schematic diagrams of edited images. Figure 17(a) shows the case where a mountain and lake landscape is the background, and Figure 17(b) shows the case where an urban landscape is the background. These edited images have the same prompt and map, but different backgrounds are generated by changing the seed value. In this way, it is possible to generate multiple patterns.
[0078] The edited images generated by the generative model are edited so that the foreground person blends naturally into the background. This is because the editing is performed accurately due to the use of maps. Furthermore, by using the person's pose information as the map, natural editing results can be obtained even in moving photographs. For example, in a scene where a person is running, the background can be changed based on the pose information, making it possible to create a natural-looking edit that reflects the movement.
[0079] This embodiment allows users to easily generate multiple editing patterns. This enables users to try different backgrounds and select the optimal editing result. Furthermore, combining the generative model and map improves the accuracy and naturalness of the editing, resulting in high-quality editing results that align with the user's intentions. Additionally, by changing the seed value, a variety of editing patterns can be generated, enabling creative editing work. Thus, this embodiment, by using prompts and maps in combination, makes it possible to generate multiple edited images that accurately reflect the user's intentions, thereby improving the efficiency and accuracy of the editing process.
[0080] Next, in step S305, the final edited image to be output is selected from the multiple edited images generated in step S304. At this time, the generated edited images are displayed on the display device 203, allowing the user to make a selection. The user is not limited to selecting only one edited image; it is possible to select multiple edited images. Alternatively, the image processing device 202 can automatically select the edited image instead of the user making a selection. Furthermore, the user may choose not to select an image in step S305 and instead select all the edited images generated in step S304. This method increases the user's freedom to select edited images according to their preferences, resulting in a more satisfying outcome. In addition, using the automatic selection function reduces the user's effort and enables efficient image processing. The ability to select multiple edited images provides different perspectives and variations, allowing for a wider range of needs to be met. This is expected to significantly improve the user experience and the flexibility of image processing.
[0081] Next, in step S306, the recording unit 202c records the edited image selected by the user in step S305 onto the recording medium 204. If multiple images were selected in step S305, all selected images are recorded. In this embodiment, since the input image is a developed image, the edited image is also a developed image. However, as in Embodiment 1, the input image may be an undeveloped image. In that case, it is necessary to add an edited image development step after the editing process execution step in step S304. Using an undeveloped image as input makes it possible to produce a higher quality edited image, but in terms of processing speed, it is more efficient to process a developed image. If an undeveloped image is used as input, the undeveloped edited image may be recorded in step S306.
[0082] Thus, using pre-developed images can improve the quality of edited images, while using developed images can improve processing speed. The choice of method should ideally be determined by considering the balance between quality and speed.
[0083] Next, in step S307, the recording unit 202c records editing information for the input image acquired in step S301. Specifically, editing information is added to the metadata of the input image. The recording of editing information is basically the same as in step S108 of Embodiment 1, but there are some important differences.
[0084] First, this embodiment differs from Embodiment 1 in that multiple edited images exist. Therefore, in order to reproduce any image, it is necessary to record editing information corresponding to multiple edited images for the input image. For example, if there is editing information 1 corresponding to edited image 1 and editing information 2 corresponding to edited image 2, then each piece of editing information is recorded. By individually saving the editing information corresponding to each edited image in this way, it becomes possible to reproduce any edited image later.
[0085] This embodiment also differs from Embodiment 1 in that it records map information (mask). If the prompt and map are common but the seeds are different, it is possible to save multiple seeds. The map itself can also be saved. For example, using a format such as TIFF, it is possible to save the map information as well. There are other formats that can save multiple images besides TIFF, so it is not limited to this. It is also possible to write the map information as Exif metadata. Alternatively, one may adopt a method in which the input image is duplicated for each edited image, and the editing information is saved for each duplicated input image (third image). This method has the advantage that the management of editing content is easier because there is an individual input image corresponding to each edited image. By using these methods, it is possible to efficiently record editing information corresponding to multiple edited images and later reproduce any edited image. Furthermore, by saving detailed editing information including map information, more accurate reproduction becomes possible, and improvements in the efficiency and quality of editing work can be expected.
[0086] In this embodiment, the control device and the image processing device may be able to communicate with each other, and the control device may send a request to the image processing device 202 regarding the execution of processing on an input image, and the image processing device may be configured to execute the image processing method of this embodiment in response to that request. Note that the control device and the image processing device are not limited to, for example, a combination of an imaging device 201 and an image processing device 202. For example, the image processing device on the cloud may be configured to execute the image processing method of this embodiment in response to a request from a user using a control device such as a personal computer or smartphone, and the control device may receive the result.
[0087] In this embodiment, by using prompts and maps in combination, it becomes possible to edit in a way that accurately reflects the user's intent, thereby improving the efficiency and accuracy of the editing process.
[0088] (Other Embodiments) The present disclosure can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. According to each embodiment, in an image generation process using a generation model, by recording editing information in the image used for image generation, it is possible to provide an image processing method, image processing device, imaging device, image processing system, and program with high reproducibility.
[0089] While preferred embodiments of this disclosure have been described above, this disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. The embodiments may also be combined.
Claims
1. An image processing method comprising the steps of: acquiring first information including a prompt for a first image; generating a second image based on the first image and the first information using a generative model; and associating the first information with a third image based on the first image.
2. The image processing method according to claim 1, characterized in that the third image is an image before it is input to the generation model.
3. The image processing method according to claim 1 or 2, characterized in that the third image is the first image, a copy of the first image, or a corrected image of the first image.
4. The image processing method according to any one of claims 1 to 3, characterized in that the prompt is information based on user instructions.
5. The image processing method according to any one of claims 1 to 4, characterized in that, in the step of associating the first information, the first information is added to the metadata of the third image.
6. The image processing method according to any one of claims 1 to 5, characterized in that the first information includes information relating to a seed number.
7. The image processing method according to any one of claims 1 to 6, characterized in that the first information includes map information.
8. The image processing method according to any one of claims 1 to 7, characterized in that the first image is an image before gamma correction.
9. The image processing method according to any one of claims 1 to 7, characterized in that the first image is an image after gamma correction.
10. The image processing method according to any one of claims 1 to 9, characterized in that the first information differs for each region in the first image or the third image.
11. The image processing method according to claim 10, characterized in that the prompt includes a first prompt and a second prompt, the first prompt is different for each region, and the second prompt is common to each of the regions.
12. The image processing method according to claim 11, characterized in that the second prompt is a negative prompt.
13. The image processing method according to any one of claims 1 to 12, characterized in that, in the step of acquiring the first information, the first information is acquired from the metadata of the third image.
14. The image processing method according to any one of claims 1 to 13, characterized in that, in the step of associating the first information, if the first information is updated, the updated first information is associated with the third image.
15. The image processing method according to any one of claims 1 to 14, characterized in that, in the step of associating the first information, the second image is associated with the third image.
16. An image processing apparatus comprising: an acquisition unit that acquires first information including a prompt for a first image; a generation unit that generates a second image based on the first image and the first information; and a recording unit that associates the first information with a third image based on the first image, wherein the second image is generated using a generation model.
17. An imaging device characterized by having the image processing device described in claim 16 and an image sensor.
18. An image processing system comprising a control device capable of communicating with each other and an image processing device according to claim 16, wherein the control device transmits a request for the execution of processing on the first image to the image processing device, and the image processing device executes the processing in response to the request.
19. A program characterized by causing a computer to execute the image processing method described in any one of claims 1 to 15.
