Program, recording medium, image processing apparatus, and image processing method

Abstract

This program is for processing illustration data having a layer structure in which image elements for forming a display image of an illustration are separated into a plurality of layers. The program causes a computer to execute: an acquisition process for acquiring illustration data to be processed; a setting process for setting a target region in a display image of the illustration data acquired in the acquisition process; a selection process for selecting two or more layers as target layers from the plurality of layers; and a generation process for, with respect to the target layers selected in the selection process, generating image elements of the target region on the basis of image elements outside the target region. The generation process includes: a first generation process for generating, on the basis of an integrated image generated by integrating the image elements of the target layers, the image elements of the target region as a reference image; and a second generation process for generating, for each target layer, the image elements of the target region corresponding to the target layer, on the basis of the reference image and the image elements outside the target region of the target layer.

Description

Program, Recording Medium, Image Processing Apparatus, and Image Processing Method

[0001] The present invention relates to a program, a recording medium, an image processing apparatus, and an image processing method, and more particularly to a technique for generating an image of a predetermined region based on surrounding images.

[0002] A two-dimensional image (illustration) of a character is separated into image elements for each part such as hair, eyebrows, eyes, and contours (face), and a curved surface corresponding to the part is applied to define a geometric deformation of the curved surface corresponding to different viewing directions from the illustration. Thus, there is a technique for generating a model that enables three-dimensional animation expressions of characters corresponding to various viewing directions (hereinafter referred to as three-dimensional expressions) (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2009-104570

[0004] By the way, in the technique described in Patent Document 1, for example, in order to realize expressions such as a change in the viewing direction or a swaying of parts, it is necessary to prepare image elements for parts that are not shown in the two-dimensional illustration (shielded on the illustration) (hereinafter referred to as shielded parts). That is, the designer needs to prepare the image elements of the parts in a state including the information of the shielded parts in advance assuming the three-dimensional expressions that can be handled by the model, or add the information of the shielded parts as needed while separating the illustration into parts.

[0005] From the viewpoint of being able to generate a model that can realize a suitable three-dimensional expression even for those with little knowledge of three-dimensional expressions, the latter method has an advantage. However, for example, it is troublesome to add the information of the shielded parts each time while defining the geometric deformation of the curved surface. In particular, when the illustration data has a structure including a plurality of layers corresponding to various effect expressions, such labor increases in proportion to the number of layers involved by the parts.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a program, a recording medium, an image processing apparatus, and an image processing method for efficiently generating image elements of a desired region by adapting them to the layer structure of illustration data.

[0007] To achieve the aforementioned objectives, the present invention provides a program for processing illustration data having a layer structure in which image elements for forming a display image of an illustration are separated into multiple layers, wherein the program causes a computer to perform the following: an acquisition process for acquiring illustration data to be processed; a setting process for setting a target area in the display image relating to the illustration data acquired in the acquisition process; a selection process for selecting two or more layers from the multiple layers as target layers; and a generation process for generating image elements of the target area for the target layers selected in the selection process based on image elements outside the target area, wherein the generation process includes a first generation process for generating image elements of the target area as a reference image based on an integrated image generated by integrating the image elements of the target layers; and a second generation process for generating image elements of the target area corresponding to each target layer based on the reference image and image elements outside the target area of ​​the target layer.

[0008] With this configuration, according to the present invention, it is possible to efficiently generate image elements of a desired region by adapting them to the layer structure of the illustration data.

[0009] A block diagram illustrating the hardware configuration of PC100 according to an embodiment and modified example of the present invention. Another diagram illustrating the division of parts in the illustration. Another diagram illustrating the division of parts in the illustration. Another diagram illustrating the division of parts in the illustration. A diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another diagram illustrating the spreading function according to an embodiment and modified example of the present invention. Another flowchart illustrating the generation process performed in PC100 according to Embodiment 1 of the present invention. A diagram illustrating the setting of the target area according to Embodiment 2 of the present invention. Another diagram illustrating the setting of the target area according to Embodiment 2 of the present invention. A diagram illustrating the generation of a reference image according to Embodiment 2 of the present invention. Another diagram illustrating the generation of a reference image according to Embodiment 2 of the present invention. Another diagram illustrating the generation of a reference image according to Embodiment 2 of the present invention. Another diagram illustrating the generation of a reference image according to Embodiment 2 of the present invention.

[0010] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the features described in the embodiments may be combined arbitrarily. Also, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0011] One embodiment described below illustrates an example of applying the present invention to a PC capable of processing illustration data having a layer structure, as an example of an image processing apparatus. However, the present invention is applicable to any device capable of processing illustration data having a layer structure.

[0012] Furthermore, in this specification, "representation model" is described as data configured to enable the three-dimensional representation of a two-dimensional image (illustration) of an object such as a character. The representation model is composed of a curved surface to which the two-dimensional image of the part is applied as a texture for each part of the object, and the deformation of each curved surface is predetermined for discrete object states. With this definition, for example, when posture information is given, the representation model can be formed by deforming each curved surface to the state corresponding to that posture and drawing it, thereby forming a two-dimensional image of the character showing the appearance of that state. Therefore, by providing a continuously changing state to the representation model, it is possible to show an animation representation as if the illustration of the object is moving in three dimensions.

[0013] 《Configuration of PC 100》 Figure 1 is a block diagram showing the hardware configuration of PC 100 according to an embodiment of the present invention.

[0014] The control unit 101 is a control device such as a CPU, and controls the operation of each block in the PC 100. Specifically, the control unit 101 reads programs related to the operating system stored in the storage device 102, programs related to illustration processing applications that process illustration data for representation models, and other such programs, expands them into the memory 103, and executes them to control the operation of each block.

[0015] The storage device 102 is, for example, a non-volatile memory such as a rewritable ROM, or an information storage device such as an HDD that is detachably connected to the PC 100. The storage device 102 may also include a recording medium such as a disk that can be accessed via a predetermined read / write interface, such as an optical drive. The storage device 102 stores not only the program described above, but also information such as parameters necessary for the operation of each block, as well as illustration data, etc.

[0016] Memory 103 is a volatile memory such as RAM. Memory 103 is used not only as an expansion area for expanding programs read from storage device 102, but also as a storage area for storing intermediate data output during the operation of each block. Alternatively, memory 103 can also be used as an area for storing intermediate calculation results or calculation results during the execution of arbitrary processing.

[0017] The drawing unit 104 is, for example, a drawing device such as a GPU. The drawing unit 104 generates a screen (image) that will be displayed in the display area of ​​the display unit 110. The screen generated by the drawing unit 104 is then displayed sequentially on the display unit 110 by the display control unit 105, which performs display control processing.

[0018] The display unit 110 may be a display device such as an LCD. In this embodiment, the display unit 110 is described as a component of the PC 100, but the implementation of the present invention is not limited to this. The display unit 110 does not need to have the same housing configuration as the PC 100, and may be an external display device that is detachably connected to the PC 100.

[0019] The operation input unit 106 is a user interface of the PC 100, such as a mouse, keyboard, pen tablet, or game controller. When the operation input unit 106 detects an operation input made to any interface, it outputs a control signal corresponding to that operation input to the control unit 101. Alternatively, the operation input unit 106 notifies the control unit 101 of the occurrence of an event corresponding to that operation input.

[0020] 《Overview of the Illustration Processing Application》 Below, an overview of the illustration processing application used in the PC 100 of this embodiment to process illustration data for use in a representation model will be described. The illustration processing application reads the illustration data to be processed and generates a group of textures from the illustration data that will be applied to the curved surfaces corresponding to each part of the representation model. That is, the illustration processing application performs a process of dividing the illustration data to be processed into images that will be applied as textures to each part of the representation model. Hereinafter, this process provided by the illustration processing application will be simply referred to as part division.

[0021] <Illustration Data to be Processed> The illustration data read by the illustration processing application of this embodiment shall have a layer structure that includes multiple layers. Illustrations related to illustration data are generally produced by sequentially adding different layers during the illustration production process. In one embodiment, for example, different layers are added and used for each task in the production process, such as line drawings and coloring (base coloring, highlight coloring, shading coloring). That is, each layer stores image information that expresses a different effect, and it is necessary to combine the image information of multiple layers in order to form the display image of the illustration.

[0022] Multiple layers have defined hierarchical relationships, and in forming a display image, the image information of lower layers is superimposed with the image information of higher layers located at the same position as the lower layer's image information. Furthermore, the manner in which the image information of each layer should be combined when forming the display image (e.g., multiplication, addition, etc.) is controlled based on information defined for each layer. In this embodiment, combining the image information of multiple layers in the combination mode defined for each layer is simply referred to as "integrating" multiple layers.

[0023] Hereinafter, image information included in a single layer will be referred to as "image elements" as elements for forming a displayed image. In contrast, image information formed by integrating image elements from multiple layers will be referred to as an "image." In this embodiment, this distinction in terminology is adopted to facilitate understanding of the invention, but both image elements and images are image information and may be interpreted interchangeably as appropriate in understanding the invention. For example, in a configuration where an image generated by integrating image elements from multiple layers is managed as a single layer, that image can be an image element (of that single layer). Also, in a configuration where a single layer is a layer group that includes lower layers, the image elements of a single layer (group) may be an image formed by integrating image elements from multiple layers.

[0024] <Part Separation> Illustration processing applications primarily provide users with various functions related to separating illustrations into parts.

[0025] The division of illustration data into parts is performed based on an operation input specifying a region within the illustration (displayed image) while displaying the displayed image of the illustration data on the display unit 110. When a region within the illustration is specified, the image information distributed within that region is separated and can be packaged as information for a single part. As described above, since the illustration data has a layer structure that includes multiple layers, the image information distributed within the specified region may consist of image elements from two or more layers. For this reason, packaging may be performed, for example, by defining a new layer group that separates the image elements distributed within the region while maintaining the layer structure of the image elements within that region. That is, if the illustration data has five types of layers, 1 to 5, and the specified region contains image elements from three of these layers, layers 1, 2, and 4, the new layer group generated by packaging will maintain the hierarchical relationship of these three types of layers and will have a configuration in which the image information contained within the region from the image elements of layers 1, 2, and 4 is stored as image elements of each layer.

[0026] Furthermore, since it would be unreasonable to expect the user to accurately input the desired area, the part division may be performed using layer masks to allow for flexible adjustment of the area afterward. For example, in the above example, for the parts to be separated, layers 1, 2, and 4 may be duplicated and added to a new layer group, and a layer mask may be applied to each of these layers to display the specified area and hide the area outside of that area, thereby performing the part division. In this case, the original layer from which the parts were separated can be reflected by applying a layer mask that hides the specified area and displays the area outside of that area.

[0027] Incidentally, the representation model is configured to show the same appearance as the illustration in the baseline state, and presents a three-dimensional representation by geometrically deforming the curved surfaces to which textures for each part are applied according to the state (hereinafter referred to simply as deforming the parts). For this reason, if parts are divided by specifying an area on the illustration, the occlusion state of the parts in the lower layer (parts distributed in the distance in the three-dimensional representation) may change due to the deformation of the parts in the upper layer (group) (parts distributed in the foreground (viewing direction) in the three-dimensional representation).

[0028] When dividing an illustration into parts, parts that are at least partially obscured by parts drawn in the foreground of the illustration (hereinafter referred to as "foreground parts") (hereinafter referred to as "background parts") will not have image information in the area where the foreground part was drawn. For example, in an illustration of a character viewed from the front, as shown in Figure 2A, the jaw of the face may be depicted in a way that obscures part of the neck. Therefore, when these are divided into parts, the part including the neck will not contain image information for the base of the neck 201, as shown in Figure 2B. Consequently, when the foreground part (face part) is deformed, the area 202 of the background part (part including the neck) that was not drawn in the illustration may be displayed, as shown in Figure 2C. That is, the area 202 may be displayed without being obscured by the foreground part. As a result, an unnatural three-dimensional representation will be displayed, in a state where the parts appear disconnected from each other.

[0029] Therefore, when dividing an illustration into parts while setting the area, it is necessary to supplement the image information of the rear part for areas where the occlusion relationship may change due to the deformation of the front part.

[0030] <Spread Fill> In this embodiment of the illustration processing application, there is a "spread fill" function that supplements the image information of areas that may be displayed due to changes in occlusion relationships caused by deformation of other parts.

[0031] The user can generate image information for a target area by setting a target area in the display image of a selected part or in the display image of an illustration (for which parts have not yet been defined) and using the fill-spread function. More specifically, the fill-spread function generates image elements for the target area such that the "fill" distributed around the target area is reflected in the target area itself. In other words, the fill-spread function generates image elements for the target area by referencing image elements outside the target area. This links the fill outside the target area with the fill inside the target area, allowing the display image to be changed to a state where the fill outside the target area is substantially spread into the target area. To facilitate understanding of the invention, the image elements outside the target area referenced in the generation of image elements for the target area may be referred to as "original elements," and the image of the target area generated by referencing the original elements may be referred to as "complementary elements."

[0032] In this embodiment of the illustration processing application, the generation of complementary elements based on the original elements is performed using inference by a trained model. This trained model is an inference model constructed by machine learning, for example, by providing images that do not have missing image information as training data, in order to infer image information for an image in which image information for a certain region is missing. In other words, this trained model is constructed to infer the image of a specified region by taking images of a specified region and the surrounding area as input. Therefore, in the fill-in function, the image elements obtained as an inference result are used as complementary elements by inputting information of the original elements and the target region into the trained model.

[0033] The trained model may be implemented as a program, for example, a function of an illustration processing application, or it may be implemented in an external device configured to communicate with a PC 100, such as an external server. In the latter case, the illustration processing application may include a program that obtains information on complementary elements, which are the inference results, from the external device by transmitting information on the original elements and target areas using an API provided by the external device for inference.

[0034] As mentioned above, illustration data can contain layers corresponding to various effect expressions, so the fills outside the target area, which are connected to the target area, can be formed by image elements from multiple layers. The image elements separated into multiple layers and included in the illustration data are designed to give different effects, such as base color fills, shading fills, and highlight fills. These image elements are important for creating the atmosphere of the original illustration in the three-dimensional representation of the representation model, and it is preferable that each is assigned a curved surface so that geometric deformations can be defined. In other words, if the original elements are provided in multiple layers around the target area, it is necessary to generate image elements for each layer in the target area while ensuring these layer structures. That is, it is necessary to generate corresponding complementary elements for each of the multiple layers that store the original elements.

[0035] Therefore, in the illustration processing application of this embodiment, the user is prompted to select as the target layer a layer containing image elements of the "fill" that they want to add to the target area using the fill-spreading function. In other words, in the fill-spreading function, the layer containing the source elements to be referenced as fills outside the target area is selected as the target layer and is referenced when generating the fill for the target area. For this reason, in the illustration processing application of this embodiment, it is possible to select one or more layers as target layers in the fill-spreading function.

[0036] If a single layer with a single color is selected as the target layer, the image elements of that target layer can be easily generated by filling the target area with pixels of the same color, thus easily achieving the desired spread of color as requested by the user. On the other hand, if two or more layers are selected as target layers, generating image elements of the target area based on the coloring of each target layer will not guarantee the occlusion relationship between the image elements of the layers in the target area, and the displayed image related to the target area and the displayed image related to the area outside the target area may not be properly linked. More specifically, since each of the two or more target layers stores image elements intended to give individual effect expressions, even if these are individually input to the trained model for inference, an image element in which the entire area of ​​the target area is filled with, for example, the effect color related to each target layer will be inferred as the complementary element corresponding to that target layer. Therefore, the complementary element of the higher layer may occlude the complementary element of the lower layer, and the displayed image related to the target area may not show the same color separation tendency as the area outside the target area. In other words, since the illustration data is composed of image elements stored in multiple layers that interact with each other to create the overall atmosphere of the illustration, simply inferring the image elements of the target area for each target layer without referencing the image elements stored in other layers does not guarantee that the same atmosphere will be created in the displayed image of the target area.

[0037] In contrast, by pre-generating an integrated image by combining the image elements of the target layer, and then using this integrated image as input to perform inference on a trained model, it is possible to generate an image of the target region that evokes a similar atmosphere to the displayed image outside the target region. That is, if the image input to the trained model integrates effect expressions related to multiple target layers, then an image containing these effect expressions for the target region can be obtained as the inference result. On the other hand, as mentioned above, in order to evoke a similar atmosphere to the illustration data in the three-dimensional representation of the representation model, it is preferable that the image elements representing each effect expression, separated into layers in the illustration data, can be edited individually. However, since the image of the target region inferred based on the integrated image is in a state where multiple effect expressions are integrated, it is not possible to edit the image elements representing each effect expression individually. In other words, the image of the target region inferred from the integrated image does not have a layer structure like the image elements that form the displayed image outside the target region (they are not separated into layers and inferred), so it is not possible to edit an image element that shows a specific effect expression independently of an image element that shows other effect expressions. Therefore, when expanding the target area, it is necessary to separate the original elements from the multiple layers containing them in the illustration data into corresponding layers, and then generate the corresponding complementary elements.

[0038] Therefore, in the illustration processing application of this embodiment, when multiple layers are selected as target layers, a unified image is first generated by integrating the image elements of the target layers, and the image obtained by inferring from this unified image to a trained model is used as a "reference image" for generating image elements of the target region. As described above, the unified image is an image that reflects the interaction of the effect expressions of each of the image elements stored in the multiple target layers, so the reference image shows the atmosphere that was created by the image elements of the target layer group outside the target region.

[0039] Therefore, by identifying the region showing the effect expression corresponding to the original element of each target layer from the reference image generated in this way, it is possible to identify the region where complementary elements corresponding to the original element of each target layer should be generated when expanding the target region. In other words, if complementary elements corresponding to a target layer are simply inferred from the original element for each target layer, the information input to the trained model is limited to the original element of one target layer, resulting in complementary elements in which image information related to that original element is distributed throughout the entire target region. However, by limiting the generation of complementary elements to the region corresponding to the original element in the reference image, complementary elements separated for each effect expression can be obtained for each target region. In other words, by going through the generation of an integrated image and the generation of a reference image, it is possible to generate complementary elements that are more convenient (separated into layers) for the target region than if complementary elements corresponding to a target layer were simply inferred from the original element for each target layer.

[0040] Here, any region segmentation method can be used for segmenting the reference image. In one embodiment, a region expansion method can be adopted in which adjacent pixels with similar pixel values ​​are included as a region. Furthermore, which segmented region in the reference image corresponds to which target layer (or its original element) can be identified, for example, based on the color information of the pixels distributed in the segmented region or the location information of the segmented region in the reference image. In one embodiment, a segmented region adjacent to (showing continuity with) a region where the original elements of a target layer are distributed can be identified as corresponding to that target layer. In another embodiment, for example, the statistical value of the color information of the pixels included in the segmented region (e.g., the average value of the pixel values) can be compared with the statistical value of the color information of the pixels included in the original elements of each target layer, and the target layer with the smallest difference can be identified as the target layer corresponding to the segmented region. Identifying the corresponding target layer based on statistical values ​​can also be done using, for example, the k-means method. Alternatively, which target layer a segmented region corresponds to may be identified using both similarity based on color information and continuity between the segmented region and its original element. However, it is easy to understand that identifying which target layer each divided region corresponds to is not limited to these methods, and other methods can be employed.

[0041] As an example of how the paint spreading function can be used, we will explain with reference to a diagram in which a target area 311 is set for a display image like Figure 3B, which is formed by image elements in a layer structure as shown in Figure 3A, and layers 301 and 302 are selected as target layers. The display image shown in Figure 3B corresponds to an image obtained by separating the image of the face portion from the character illustration shown in Figure 2A, and then adding a line drawing (image element of layer 303) that represents the outline of the neck to the display image of the remaining part including the neck. In the example of Figure 3B, the target area 311 is set along the line drawing that represents the outline of the neck and includes some of the image elements of layers 301 and 302, but the implementation of the present invention is not limited to this. The target area can be set in the display image relating to the illustration data, and it is not necessary for it to be set in a way that overlaps with the image elements of any of the layers.

[0042] At this time, first, the image elements (original elements) of the layers 301 and 302 selected as the target layers are integrated to generate an integrated image 321 as shown in FIG. 3C. When the integrated image 321 and the information of the target region 311 are input into the learned model, a reference image 331 as shown in FIG. 3D is output as an inference result. As shown in the figure, the reference image 331 reflects the characteristics of the distribution patterns of the original elements of the layers 301 and 302 in the integrated image 321, and when arranged in the target region as shown in FIG. 3E, it shows continuity with the original elements of the layers 301 and 302 outside the target region.

[0043] Here, by dividing the reference image 331 into regions, the reference image 331 can be divided into a divided region 332 corresponding to the layer 301 and a divided region 333 corresponding to the layer 302 as shown in FIG. 3D. Thereby, for the target region, a complementary element showing painting restricted to the region of the divided region 332 for the original element of the layer 301 and a complementary element showing painting restricted to the region of the divided region 333 for the original element of the layer 302 may be generated.

[0044] In one aspect, the generation of each complementary element is performed by extracting pixels corresponding to the divided region 332 from an image 341 of the target region size inferred by the learned model with the original element of the layer 301 as an input, and also by extracting pixels corresponding to the divided region 333 from an image 342 of the target region size inferred by the learned model with the original element of the layer 302 as an input, as shown in FIG. 3F. Here, the complementary element 343 generated by extraction from the image 341 and the complementary element 344 generated by extraction from the image 342 can form a display image similar to the reference image 331 shown in FIG. 3D by being displayed together.

[0045] In another embodiment, each interpolation element may directly generate image interpolation elements 343 and 344 by setting the region for which the trained model is to perform inference to the respective divided regions 332 and 333 identified based on the reference image 331. That is, in this embodiment, the target region 311 is set, but when generating interpolation elements 343 and 344 based on the reference image 331, the trained model is input with information on the original elements of the target layer and the corresponding divided regions (332 or 333).

[0046] The complementary elements 343 and 344 generated in this way are stored in separate new layers so that they can be individually edited, while remaining distinct from the image elements of layers 301 and 302 that were originally included in the illustration data. That is, by performing the fill-in operation on the target area 311, the layer structure of the display image of the part including the neck will be such that, as shown in Figure 3G, complementary element 343 is stored in layer 351, which is different from layer 301, and complementary element 344 is stored in layer 352, which is different from layer 302. This provides versatility that allows for flexible handling, for example, if the intended configuration is not inferred by the user, or if the user wants to edit the filled-in image elements individually.

[0047] 《Generation Process》 The generation process performed in the PC 100 of this embodiment for generating complementary elements for the target area will be described below using the flowchart in Figure 4. The process corresponding to the flowchart can be realized by the control unit 101 reading the corresponding processing program stored in the storage device 102, for example, loading it into the memory 103 and executing it. This generation process will be described as starting, for example, when it is detected that a target area has been set in the display area related to the loaded illustration data and that an operation input related to the execution of the fill-spreading function has been made.

[0048] In S401, the control unit 101 selects a target layer. In the generation process of this embodiment, it will be described that the target layer is selected based on an operation input related to layer selection by the user, but the implementation of the present invention is not limited to this. For example, in a mode where layer groups are provided for each part in the illustration data, the layers included in the corresponding layer group may be selected by default based on the selection status of the parts.

[0049] In S402, the control unit 101 determines whether the number of target layers is plural. If the control unit 101 determines that the number of target layers is plural, the process proceeds to S403, and if it determines that the number is not plural (i.e., one), the process proceeds to S407.

[0050] In S403, the control unit 101 generates an integrated image by integrating the image elements (original elements) of the target layers.

[0051] In S404, the control unit 101 generates a reference image based on the integrated image. Specifically, the control unit 101 uses, as the reference image, the image obtained as an inference result by inputting the integrated image and the information of the target area into a learned model.

[0052] In S405, the control unit 101 divides the reference image into regions.

[0053] In S406, the control unit 101 generates image elements (complementary elements) of the target areas corresponding to each of the target layers based on the information of the result of the region division in S405. For example, the control unit 101 sequentially selects the target layers, and extracts the pixels at the positions corresponding to the divided regions corresponding to the layer from the image obtained as an inference result by inputting the original elements of the layer and the information of the target area into a learned model, so as to generate the complementary elements corresponding to the layer.

[0054] On the other hand, when it is determined in S402 that the number of target layers is not plural, in S407, the control unit 101 generates image elements (complementary elements) of the target area corresponding to the target layer. Specifically, the control unit 101 uses the image obtained as an inference result by inputting the original elements of the target layer and the information of the target area into a learned model as the complementary elements of the target area corresponding to the target layer.

[0055] In S408, the control unit 101 adds a new layer to the illustration data and stores the generated complementary elements to complete the generation process. If complementary elements are generated for multiple target layers, the control unit 101 stores each of them in a different layer in this step. At this time, the control unit 101 stores the complementary elements in layers according to the hierarchical relationship between the multiple target layers of the target layer corresponding to the complementary element.

[0056] As described above, the image processing apparatus of this embodiment can efficiently generate image elements of a desired region by adapting them to the layer structure of the illustration data.

[0057] In the embodiments illustrated using Figures 3A to 3G, the complementary elements were described as being generated for each target layer, limited to the corresponding divided regions in reference image 331. However, considering the versatility of editing the expanded image elements, the method of generating the complementary elements can also be modified as follows.

[0058] For example, if the generated interpolation element 343 does not show the desired fill, it is assumed that the user will edit the layer 351 after generation. In this case, if editing is performed to delete the image information of a part of the interpolation element 343 (deletion of the fill), a gap will be created in the target area 311 where there is no fill. That is, in the embodiments shown in Figures 3A to 3G, the interpolation element 344 generated for the original element of layer 302 is generated in an area that does not overlap with the interpolation element 343. Therefore, if a part of the interpolation element 343 is deleted in layer 351, there will be no fill in the same position in the lower layer 352, and it may appear as a gap. In such a case, the user needs to edit the image elements of layer 352 to a state without distortion after editing the image elements of layer 351.

[0059] Therefore, for all but the topmost layer among the selected target layers, complementary elements for that layer may be generated in the region shown by adding the division regions corresponding to all layers above that layer to the division region corresponding to that layer. That is, in the example shown in Figure 3F, both the complementary element 343 corresponding to layer 301 and the complementary element 344 corresponding to layer 302 are limited to the division region 332 or 333 shown in reference image 331, but for the complementary element corresponding to layer 302, which is not the topmost layer, the complementary element is generated in the region obtained by adding division region 332 to division region 333. In the example shown in Figure 3D, since there are two types of target layers, the region in which complementary elements should be generated for layer 302 is the region obtained by adding division region 332 to division region 333 in reference image 331, i.e., the target region (the entire region of reference image 331).

[0060] In this case as well, the complementary element corresponding to layer 302 is obscured by the filling of the complementary element 343 corresponding to the upper layer 301, and as a result, the same filling spread as in Figure 3E can be achieved for the target area. In this embodiment, no gaps are created regardless of how the complementary element 343 of layer 351 is edited, improving the versatility of editing.

[0061] [Modification 1] In the embodiments described above, a configuration in which all original elements of the target layer are "fills" has been described, but the implementation of the present invention is not limited to this. The target layer may also include layers in which the stored image elements are "line drawings". In this configuration, the reference image inferred by the trained model based on the integrated image also includes line drawing image elements. Therefore, in the region segmentation of the reference image, the region in which pixels corresponding to line drawings are distributed is identified as the segmented region corresponding to the target layer of line drawings. In this case, complementary elements may be generated by extracting pixels of the segmented region corresponding to the layer from an image of the target region size inferred using the original elements of the target layer of line drawings as input, or the group of line drawing pixels extracted from the reference image itself may be used as the complementary elements of the layer in the region segmentation of the reference image.

[0062] [Embodiment 2] In Embodiment 1 described above, an example was explained in which the present invention was applied to the generation of complementary elements when setting a target area and spreading it out. However, the implementation of the present invention is not limited to such spreading applications triggered by the setting of a target area, but may also be implemented in other applications triggered by the setting of an area.

[0063] For example, as described above, the division of illustration data into parts is performed by specifying a region 501 in the display image of the illustration data, as shown in Figure 5A, and separating the image elements distributed within that region into a new layer group. When the image elements of region 501 are separated into other layer groups, the pixel information that was distributed in that region is excluded (cut out) from the original display image, so the images of parts 502, 503, and 504 distributed around region 501 become fragmented, as shown in Figure 5B. In the example shown, region 501 is set to surround the facial parts in a bust-up illustration of a character, and the image elements related to those parts are separated into a different layer, resulting in the image of the right ear part 502, the image of the left ear part 503, and the image of the torso part 504 including the neck being fragmented and not continuous.

[0064] When the representation model is generated reflecting this part division, the layer groups of parts 502, 503, and 504 become lower layer groups than the face part, so the occlusion relationship may change due to geometric deformation of the curved surface related to the face part. Therefore, for parts 502, 503, and 504, by setting target areas in region 501 according to the shape of each part and performing painting and spreading, complementary elements for each part can be generated as shown in Figure 5C. This makes it possible to adjust the images of each part so that they show a natural three-dimensional representation even when geometric deformation occurs on the curved surface to which the face part is applied.

[0065] On the other hand, repeatedly setting a target area with a shape suitable for each part separated by the cutout of area 501 and issuing a paint-spreading instruction is time-consuming, especially when there are many separated parts. For this reason, when an area is specified to be cut out from a part of the displayed image for part separation, that area may be reused as the target area for paint-spreading.

[0066] In this case, from the viewpoint of displaying parts 502, 503, and 504 in a suitable occlusion state even when geometric deformation occurs on the curved surface to which the facial parts are applied, adding image information to the entire region 501 and spreading it to connect these parts is undesirable. That is, in the manner in which complementary elements of the shape of region 501 are added to connect parts 502, 503, and 504, adjustment (geometric modification) of the added complementary elements and the curved surface to which parts 502, 503, and 504 are applied is necessary each time geometric deformation occurs on the curved surface to which the facial parts are applied. Therefore, it is difficult to create a representation model that can realize a general-purpose three-dimensional representation. For this reason, in the spreading function of this embodiment, when cropping of the displayed image occurs due to part division, a method will be described in which complementary elements with shapes suitable for each of the parts distributed around the target area are generated, with the cropped area as the target area. That is, when cropping occurs due to part division, a method will be described in which complementary elements with shapes suitable for each part are generated all at once in the cropped area (target area), similar to Figure 5C.

[0067] By configuring the representation model to have curved surfaces only for parts of objects such as characters, it is possible to achieve a display that looks natural even when the generated three-dimensional image is placed (superimposed) on a desired background. In other words, the representation model only needs to have curved surfaces for the significant pixels of the parts, and the illustration data used to generate it is configured so that pixels that do not correspond to parts do not have image information. Therefore, in the illustrations related to the illustration data, pixels that do not have image information are set as transparent pixels without color information.

[0068] In this case, generating suitable complementary elements for the target area requires a process different from the spreading method described in Embodiment 1.

[0069] The spread-over method described in Embodiment 1 involved inputting a unified image, which was created by integrating the original elements of multiple target layers, into a trained model to obtain an inference result for a reference image that filled all the pixels of the target region, and then generating complementary elements corresponding to each target layer based on the information of each segmented region of the reference image. In other words, in the spread-over method, all pixels of the target region belong to a segmented region corresponding to one of the target layers, so as a result, complementary elements are generated for the entire target region.

[0070] In contrast, in the embodiment where the target area is the cut-out region related to part division, it is preferable that only complementary elements with shapes adapted to each of the divided parts are generated, and the rest are set as transparent pixels. Therefore, some pixels in the target area need to be classified as divided areas that do not correspond to any target layer. In other words, in the embodiment where the target area is the cut-out region, it is preferable that the reference image based on the integrated image is generated to include not only divided areas corresponding to the original elements of the target layer, but also divided areas corresponding to transparent pixels.

[0071] To obtain such reference images, in the embodiment where the cropped area is the target area, transparent pixels are also converted into information of significant pixels to generate an integrated image. That is, when generating an integrated image to be input to the trained model, not only the image elements (original elements) of the target layer are included, but also pixels that are transparent in all target layers (pixels that do not contain significant pixel information in any target layer). Here, transparent pixels have an α value of 0 and do not have any color information, but when generating an integrated image, the α value is set to 1 and a pixel value indicating a predetermined color (for example, white) is assigned to them.

[0072] For example, consider the case shown in Figure 5B, where an illustration with facial features cut out is selected as the target layer, and the line drawing and color layers constituting the illustration are selected. In this case, if all target layers are merged without including transparent pixels, an image is generated in which the hatched areas do not have pixel values, as shown in Figure 6A. Conversely, if all target layers are merged after converting transparent pixels to colored pixels representing white, an image is generated in which the areas without pixel values ​​are not included, as shown in Figure 6B.

[0073] Here, the predetermined color is set to identify pixels that should be recognized as transparent pixels in subsequent image processing, and is not limited to white. The predetermined color can be selected from colors that are different from any of the pixels that make up the illustration in the illustration data.

[0074] Therefore, by inputting a unified image generated by converting transparent pixels in all target layers to pixels of a predetermined color into a trained model, it is possible to infer an image containing pixels of the predetermined color for the target region. Then, by changing the pixels of the predetermined color in this image to transparent pixels, a reference image can be generated. Furthermore, by segmenting the obtained reference image into regions, it is possible to generate image elements (complementary elements) corresponding to each target layer for each part distributed around the target region, based on the segmented regions distributed in the vicinity.

[0075] For example, by using the integrated image in Figure 6B as input to a trained model for inference, an image like that in Figure 6C can be obtained for the target region. Then, by changing the pixels of a predetermined color in that image to transparent pixels, a reference image like that in Figure 6D can be obtained. In the example in Figure 6D, the reference image allows for the identification of the division regions of line drawings and fills related to the complementary elements of part 502, from image element 601 in the reference image distributed near part 502; the division regions of line drawings and fills related to the complementary elements of part 503, from image element 602 in the reference image distributed near part 503; and the division regions of line drawings and fills related to the complementary elements of part 504, from image element 603 in the reference image distributed near part 504.

[0076] By doing this, when a cropping area is specified in the displayed image for part division, image elements (complementary elements) with shapes suitable for each of the parts distributed around that area can be generated for the cropping area, thereby reducing the effort required from the user.

[0077] [Modification 2] In Modification 1 described above, when line drawings and fills are selected as target layers, a method is described in which a segmented area corresponding to the line drawing is extracted from a reference image inferred based on an integrated image obtained by integrating the image elements of these target layers, and complementary elements for the line drawing in the target area are generated. On the other hand, in an integrated image that includes line drawings, inference is performed by treating the line drawing as one aspect of the fill, so the inference result may not necessarily capture the unique characteristics of the line drawing.

[0078] Line drawings are often used in illustrations to represent the edges and boundaries of filled areas. Therefore, the complementary elements (line drawings) of a target region inferred by a trained model based on the original elements of the target layer containing the line drawings may contain useful information in terms of identifying the areas where complementary elements for filled areas should be generated. In particular, when generating complementary elements for cut-out regions related to part division, referring to the unique features of the line drawings may allow for the optimal identification of the shape of the complementary elements for filled areas within the cut-out region.

[0079] Therefore, when generating complementary elements for a target area, it is possible to determine whether the original elements of the target layer are line drawings or filled images, and to perform some processing differently depending on the determination result. Here, whether the image elements (original elements) stored in the target layer are line drawings or not may be determined, for example, by referring to the attribute information of the target layer set by the user, or by performing image analysis on the image elements of each target layer. In the case of image analysis, for example, a skeleton image of the image elements of the target layer can be generated using the Zhang-Suen thinning algorithm, and the determination can be made based on the ratio of the skeleton length (number of pixels) in the skeleton image to the area (number of pixels) occupied by the image element.

[0080] If the image elements stored in the target layer are line drawings, an image in which line drawings are placed within a predetermined color region can be generated by converting the transparent pixels of the target layer to pixels of a predetermined color, similar to Embodiment 2. Therefore, by inputting this image into a trained model, complementary elements of the line drawing can be inferred for the target region. The complementary elements of the line drawing obtained by this inference may exhibit line drawing-specific features more than a reference image obtained based on an integrated image that combines line drawing and colored image elements.

[0081] On the other hand, for the target region, the inference results based solely on the original line drawing elements and the inference results based on a combined image that integrates the original line drawing and fill elements differ because the characteristics of the input images are different. As a result, the line drawing complement elements for the target region based on the former inference result and the fill complement elements for the target region based on the latter inference result may not correspond (the region defined by the line drawing and the region occupied by the fill image elements may not match).

[0082] To resolve such discrepancies between complementary elements, the group of complementary elements generated for the target area may be further adjusted. In one embodiment, complementary elements based solely on line drawings may be prioritized, and the complementary elements for fills may be adjusted to fit these complementary elements. Such adjustments may include, for example, deleting fills that extend beyond the area defined by the complementary elements based solely on line drawings, or supplementing missing pixels in the area up to the boundary of the area by duplicating pixels from the complementary elements for fills. In another embodiment, the complementary elements for the final line drawing may be generated by weighting and adding the complementary elements based solely on line drawings and the complementary elements for line drawings identified based on a reference image. In this case as well, the complementary elements for fills may be adjusted to fit the complementary elements for the final line drawing.

[0083] [Other Embodiments] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention.

[0084] 100: PC, 101: Control Unit, 102: Storage Unit, 103: Memory, 104: Drawing Unit, 105: Display Control Unit, 110: Display Unit, 106: Operation Input Unit

Claims

1. A program for processing illustration data having a layer structure in which image elements for forming a display image of an illustration are separated into multiple layers, wherein the program causes a computer to perform the following: an acquisition process for acquiring the illustration data to be processed; a setting process for setting a target area in the display image relating to the illustration data acquired in the acquisition process; a selection process for selecting two or more layers from the multiple layers as target layers; and a generation process for generating image elements of the target area for the target layers selected in the selection process based on image elements outside the target area, wherein the generation process includes: a first generation process for generating image elements of the target area as a reference image based on an integrated image generated by integrating the image elements of the target layers; and a second generation process for generating image elements of the target area corresponding to each of the target layers based on the reference image and image elements outside the target area of ​​the target layer.

2. The program according to claim 1, wherein the generation process further includes a division process for dividing the reference image into regions, and the second generation process generates image elements of the target region corresponding to the target layer for each of the target layers, based on region information indicating the region in the reference image corresponding to the target layer.

3. The program according to claim 2, wherein the second generation process generates image elements of the target region corresponding to the target layer by extracting an image of the region indicated by the region information from an intermediate image of the target region generated based on image elements outside the target region of the target layer for each of the target layers.

4. The program according to claim 2, characterized in that the second generation process generates an image of the region indicated by the region information based on the image elements outside the target region of the target layer for each of the target layers, thereby generating an image element of the target region corresponding to the target layer.

5. The program according to claim 2, wherein a hierarchy is defined among the multiple layers, and the second generation process generates an image element of the target region corresponding to the first layer by extracting an image of the region from an intermediate image of the target region generated based on the image elements outside the target region of the first layer, from the region indicated by the region information of the first layer plus the region indicated by the region information of a second layer which is higher than the first layer, for the first layer among the target layers.

6. The program according to claim 2, wherein a hierarchy is defined among the multiple layers, and the second generation process generates an image of the region corresponding to the first layer by adding the region indicated by the region information of a second layer that is higher than the first layer to the region indicated by the region information of the first layer, based on the image elements outside the target region of the first layer, for the first layer among the target layers.

7. The program according to any one of claims 2 to 6, characterized in that the generation process further includes a identification process that identifies the target layer corresponding to each region obtained by dividing the reference image and outputs the region information.

8. The program according to claim 7, characterized in that the identifying process identifies the target layer corresponding to the region based on at least one of the color distribution of each region obtained by dividing the reference image and the continuity with the image elements of the target layer.

9. The program according to any one of claims 1 to 8, characterized in that the program causes the computer to further perform a separation process to separate the image elements of a partial region in the displayed image relating to the illustration data into a new layer, and when the separation process is performed, the setting process sets the partial region as the target region, and the first generation process, when the integrated image includes transparent pixels, changes the transparent pixels to pixels of a predetermined color, and generates the reference image by changing the pixels of the predetermined color to transparent pixels from the image elements of the target region generated based on the modified integrated image.

10. The program according to any one of claims 1 to 9, characterized in that the second generation process generates image elements of the target region corresponding to each of the target layers as image elements of a layer different from the target layer.

11. The program according to any one of claims 1 to 9, characterized in that the generation process generates image elements of the target region by having a trained model, which is constructed to infer an image of the specified region based on images surrounding the specified region, infer image elements outside the target region as input.

12. A computer-readable recording medium having the program described in any one of claims 1 to 11 recorded on it.

13. An image processing apparatus for processing illustration data having a layer structure in which image elements for forming a display image of an illustration are separated into multiple layers, comprising: an acquisition means for acquiring the illustration data to be processed; a setting means for setting a target area in the display image relating to the illustration data acquired by the acquisition means; a selection means for selecting two or more layers from the multiple layers as target layers; and a generation means for generating image elements of the target area for the target layers selected by the selection means based on image elements outside the target area, wherein the generation means comprises: a first generation means for generating image elements of the target area as a reference image based on an integrated image generated by integrating the image elements of the target layers; and a second generation means for generating image elements of the target area corresponding to each of the target layers based on the reference image and image elements outside the target area of ​​the target layer, the image processing apparatus characterized by this, comprising 14. An image processing method for processing illustration data having a layer structure in which image elements for forming a display image of an illustration are separated into multiple layers, comprising: an acquisition step of acquiring the illustration data to be processed; a setting step of setting a target area in the display image relating to the illustration data acquired in the acquisition step; a selection step of selecting two or more layers from the multiple layers as target layers; and a generation step of generating image elements of the target area for the target layers selected in the selection step based on image elements outside the target area, wherein the generation step comprises: a first generation step of generating image elements of the target area as a reference image based on an integrated image generated by integrating the image elements of the target layers; and a second generation step of generating image elements of the target area corresponding to each of the target layers based on the reference image and image elements outside the target area of ​​the target layer.

Images