Information processing device, information processing method, and program

Abstract

An information processing device that: acquires image data of a person from whom a target object is derived; generates a color map, which is a texture map representing the color of the appearance of the target object, on the basis of the acquired image data of the person; and generates a subcutaneous tissue map, which is a texture map for determining the effect of the person's subcutaneous tissue on the appearance of the target object, on the basis of the acquired image data of the person.

Description

Information Processing Apparatus, Information Processing Method, and Program

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program used for creating model data of a three-dimensional object used for drawing an image including a human skin.

[0002] In the field of 3D computer graphics, various devices have been made to draw images with high reality. As a result, in recent years, it has become possible to draw images closer to real photos.

[0003] As one of the methods for expressing the human skin in the above-described technology, a method of expressing the influence of subcutaneous tissue such as redness on the human skin by using a texture map prepared separately from the texture map representing the appearance of a person is being studied. In this method, for example, a texture map for expressing redness is prepared in advance, and when redness appears on the skin of the target person, this texture map for expressing redness is synthesized with the texture map of the appearance, etc., to express the state where redness has occurred on the human skin. As a result, it becomes possible to easily draw both the normal state and the state where redness has occurred on the skin.

[0004] In order to use such a method, it is necessary to generate in advance a texture map for expressing the influence of subcutaneous tissue separately from the texture map of the appearance. In order to generate such a texture map, a method such as measuring how the subcutaneous tissue of the target person is distributed using a dedicated measuring device can be considered. As the dedicated measuring device in this case, the existence of a measuring device for measuring the state of a person's subcutaneous tissue for purposes such as medical treatment and beauty is known. However, the method of using such a dedicated measuring device is laborious and costly.

[0005] The present invention has been made in consideration of the above circumstances, and one of its purposes is to provide an information processing apparatus, an information processing method, and a program that can relatively easily create model data capable of expressing the influence of subcutaneous tissue on human skin.

[0006] An information processing device according to one aspect of the present invention is an information processing device that includes: an image data acquisition means for acquiring image data of a person that is the basis of a target object; a color map generation means for generating a color map, which is a texture map representing the color of the appearance of the target object, based on the acquired image data of the person; and a subcutaneous tissue map generation means for generating a subcutaneous tissue map, which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object, based on the acquired image data of the person.

[0007] An information processing method according to one aspect of the present invention is an information processing method that includes the steps of: acquiring image data of a person that will be the basis of a target object; generating a color map, which is a texture map representing the color of the appearance of the target object, based on the acquired image data of the person; and generating a subcutaneous tissue map, which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object, based on the acquired image data of the person.

[0008] A program according to one aspect of the present invention is a program that causes a computer to perform the following steps: acquiring image data of a person that will be the basis of a target object; generating a color map, which is a texture map representing the color of the appearance of the target object, based on the acquired image data of the person; and generating a subcutaneous tissue map, which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object, based on the acquired image data of the person. This program may be provided stored in a computer-readable, non-temporary information storage medium.

[0009] This is a configuration block diagram showing the configuration of an information processing device according to an embodiment of the present invention. This is a functional block diagram showing the functions of an information processing device according to an embodiment of the present invention. This is a diagram showing an example of the processing flow executed by an information processing device according to an embodiment of the present invention when creating model data.

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0011] Figure 1 is a block diagram showing the configuration of an information processing device 1 according to one embodiment of the present invention. The information processing device 1 is a personal computer, a server computer, etc., and as shown in the figure, it is configured to include a control unit 11, a storage unit 12, and an interface unit 13. The information processing device 1 is also connected to a display device 14 and an operating device 15.

[0012] The control unit 11 includes at least one processor such as a CPU and performs various information processing according to the program stored in the storage unit 12. Specific examples of the processing performed by the control unit 11 in this embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM and stores the program executed by the control unit 11 and the data processed by the program.

[0013] The interface unit 13 is an interface for data communication between the display device 14 and the operating device 15. The information processing device 1 is connected to the display device 14 and the operating device 15 via the interface unit 13 by either wired or wireless means. Specifically, the interface unit 13 includes a multimedia interface for transmitting video data supplied by the information processing device 1 to the display device 14. It also includes a data communication interface for receiving signals indicating the operations performed by the user on the operating device 15.

[0014] The display device 14 is a liquid crystal display, an organic EL display, or the like, and displays an image on the screen corresponding to the video signal supplied from the information processing device 1. The operating device 15 is, for example, a keyboard or mouse, and accepts user input.

[0015] The functions implemented by the information processing device 1 will be explained below using the functional block diagram in Figure 2. In this embodiment, the information processing device 1 generates data for three-dimensional objects (hereinafter referred to as model data M) used to draw and present to the user the appearance of a virtual three-dimensional space (virtual space). In particular, for an object of a specific person to be drawn, the information processing device 1 generates the model data M necessary to draw that person's object based on the original person's image data.

[0016] In the following, the human object drawn by the model data M output by the information processing device 1 will be referred to as the target object. In this context, the target object to be drawn will specifically be the head object, including the human face. However, the information processing device 1 according to this embodiment may also output model data M for other parts of the human body other than the head using the same method. The model data M of the target object includes at least shape data that defines the general shape of the target object and appearance data used to determine the appearance of the surface of the target object. Furthermore, as will be described later, the appearance data includes a plurality of texture map data that are mapped to the surface of the target object.

[0017] As shown in Figure 2, the information processing device 1 functionally includes a person image acquisition unit 21, a color map generation unit 22, a redness map generation unit 23, and a model data output unit 24. These functions are realized by the control unit 11 operating according to a program stored in the storage unit 12. This program may be provided to the information processing device 1 via a communication network such as the Internet, or it may be provided stored on a computer-readable information storage medium such as an optical disc.

[0018] The person image acquisition unit 21 acquires image data of a person that will serve as the basis for the target object. Specifically, for example, the person image acquisition unit 21 acquires image data of a real person by capturing images of that person and processing them as necessary. Hereinafter, the image data of a person acquired by the person image acquisition unit 21 will be referred to as person image data I. Here, in order to generate model data M that can draw a three-dimensional object, the person image data I may be data that includes multiple planar images obtained by capturing the same person from various angles, such as the front, left side, and right side.

[0019] The color map generation unit 22 generates a texture map containing color information of the target object's appearance based on the person image data I acquired by the person image acquisition unit 21. Hereinafter, the texture map containing the target object's color information will be referred to as the color map C.

[0020] Here, the color map C has a data structure similar to planar image data, and is data in which a pixel value representing a color in a given color space is set for each of multiple pixels arranged along a two-dimensional coordinate system (e.g., UV coordinate system). Each pixel included in the color map C is associated with a specific position on the surface of the target object.

[0021] When a target object is drawn using the model data M generated by the information processing device 1 according to this embodiment, texture mapping is performed on the surface of the target object placed in the virtual space based on this color map C. This determines the color of the target object's surface. The process by which the color map generation unit 22 generates the color map C may be based on known methods.

[0022] In this example, the color map generation unit 22 generates a color map C that defines the color of the target object surface. However, it is not limited to this; it may also generate other map data to be included in the model data M, such as a height map that represents the height of the object surface.

[0023] The redness map generation unit 23 generates a texture map used to represent the state of redness in human skin, based on the human image data I acquired by the human image acquisition unit 21. Hereinafter, the texture map generated by the redness map generation unit 23 to represent redness will be referred to as the redness map R.

[0024] The redness map R is used to represent, for example, the reddish appearance of skin on a person's cheeks or forehead. However, since the redness of the skin represented by this redness map R is temporary and caused by factors such as cold or excitement, it is not possible to include such redness representation in the color map C that is always used when rendering the target object. Therefore, in this embodiment, a redness map R representing the redness appearing on the target object's skin is generated separately from the target object's color map C and included in the target object's model data M.

[0025] The redness map R is a texture map that targets the same area of ​​the target object surface as the color map C generated by the color map generation unit 22, and may have the same data structure as the color map C. In this case, the redness map R is composed of pixels arranged along the same coordinate system as the color map C, and the pixel value of each pixel is set to a value representing the redness color that appears on the skin of a person at the position corresponding to that pixel.

[0026] One of the distinctive features of this embodiment is that the redness map R is generated based on the same person image data I used to generate the color map C. That is, based on a single person image data I, both a color map C representing the normal color of the same person and a redness map R representing the redness that occurs in the person's skin are generated.

[0027] Generally, accurately identifying the location and degree of redness in a person's skin requires methods such as measuring the structure of the subcutaneous tissue inside the skin using specialized measuring equipment. However, generating a redness map R using measurement results obtained by such methods is time-consuming and costly. Therefore, in this embodiment, a redness map R is generated by estimating where and to what degree redness occurs in the skin based on person image data I representing the appearance of the target person. This makes it relatively easy to generate a set of color map C and redness map R for the same person.

[0028] The redness map generation unit 23 generates a redness map R using an image based on the person image data I as an input image. Here, the image based on the person image data I may be the image represented by the person image data I itself, or it may be an image obtained by performing a predetermined image processing on the person image data I. Alternatively, the redness map R may be generated using a color map C generated by the color map generation unit 22 from the person image data I as an input image. In any case, the redness map R is generated directly or indirectly based on the person image data I. That is, the pixel value of each pixel included in the redness map R is determined based on the color information included in the person image data I. Therefore, there is no need to perform measurements using separate measuring equipment for the generation of the redness map R.

[0029] Specifically, for example, the redness map generation unit 23 performs predetermined image processing to estimate the state of the subcutaneous tissue for each pixel representing human skin in the input image. Then, it generates a redness map R using this estimation result. As a concrete example, such image processing may be a process that performs multivariate analysis, such as principal component analysis or independent component analysis, on the pixel values ​​in a predetermined color space of the target pixel and / or surrounding pixels.

[0030] Furthermore, the redness map generation unit 23 may generate the redness map R using a pre-trained model generated by machine learning. Specifically, the redness map generation unit 23 inputs an input image based on human image data I to a pre-prepared pre-trained model. The map data output by this pre-trained model is then adopted as the redness map R.

[0031] Here, we will explain a specific example of a method for generating such a pre-trained model. This pre-trained model may be generated by machine learning using image data obtained by imaging a real person and measurement data obtained by measuring the subcutaneous tissue of that person. As mentioned above, by performing measurements using specialized measuring equipment, it is possible to generate a redness map R with relatively high accuracy for a real person. Therefore, we will perform machine learning using image data obtained by imaging multiple people as input data, and a redness map R generated based on measurement data obtained by measuring the same person with specialized measuring equipment as the ground truth data. This makes it possible to generate a pre-trained model that takes image data of a person as input data and the redness map R of that person as output data. By inputting an input image of an unknown person to the machine learning model obtained in this way, it is possible to generate a redness map R for that person without performing measurements using measuring equipment on the person themselves.

[0032] The model data output unit 24 generates and outputs the final model data M of the target object. This model data M includes at least shape data that defines the shape of the target object in virtual space, a color map C generated by the color map generation unit 22, and a red map R generated by the red map generation unit 23.

[0033] The image processing device, utilizing the model data M obtained in this manner, first places the target object in the virtual space and then texture-maps the color map C onto its surface to represent the appearance of the person when generating an image of the virtual space in which the target object appears. Furthermore, in situations where redness is expected to occur in the person's skin, the redness of the person's skin is represented by texture-mapping a redness map R onto the color map C. This makes it possible to achieve a more realistic depiction of the person. In this case, the image processing device may, depending on the situation, map only a portion of the redness map R (for example, only the area corresponding to the person's cheeks) rather than the entire redness map R. Alternatively, the pixel values ​​included in the redness map R may be corrected before mapping depending on the degree of redness in the skin.

[0034] More specifically, an image processing device using model data M calculates a value for each location of the target object that indicates the extent to which the pressure affects the skin, based on factors such as the area and magnitude of the pressure, in situations where pressure is applied to the skin on the surface of the target object. Then, based on these calculated values ​​indicating the degree of influence from external factors, the pixel values ​​included in the redness map R are corrected, and mapping is performed. Furthermore, if such external factors persist, the correction value indicating the degree of influence may be varied over time, and mapping may be performed again. This makes it possible to represent the dynamic changes that occur in human skin over time, even in a virtual space.

[0035] Here, using Figure 3, we will explain an example of the processing flow (pipeline) that the information processing device 1 executes based on the creator's instructions when the creator creates the model data M of the target object.

[0036] Although omitted in the above explanation, the information processing device 1 acquires shape recognition data to identify the three-dimensional shape of the target person, along with person image data I, in order to generate shape data for the target object. Based on this shape recognition data, the information processing device 1 performs a modeling process to determine the shape of the target object and outputs it as shape data (S1). This shape data may be data composed of multiple meshes, for example.

[0037] Furthermore, the color map generation unit 22 of the information processing device 1 generates a color map C based on the shape data and the person image data I (S2). Also, the redness map generation unit 23 generates a redness map R based on the shape data and the person image data I (S3).

[0038] Subsequently, the model data output unit 24 merges the shape data generated in S1, the color map C generated in S2, and the redness map R generated in S3 (S4). Then, it adds control data necessary to control the model data M, such as data for rigs and blend shapes (S5). This completes the model data M that can finally be used by other image processing devices.

[0039] As described above, according to the information processing device 1 of this embodiment, by generating both a color map C and a redness map R from the same person image data I, model data M of a target object capable of representing the influence of the subcutaneous tissue of human skin can be generated relatively easily through a series of pipeline processes.

[0040] It should be noted that the embodiments of the present invention are not limited to those described above. For example, in the above description, in addition to the color map C, a redness map R is generated to represent the redness that occurs on human skin. However, the invention is not limited to this, and map data for various texture maps (subcutaneous tissue maps) to represent the effects that occur on the surface of human skin due to subcutaneous tissue may be generated based on the person image data I and included in the final output model data M. For example, the information processing device 1 may generate, in place of or in addition to the redness map R, a sebum secretion amount map showing the degree of sebum secretion of the skin, or a moisture content map showing the degree of dampness and hydration of the skin, as subcutaneous tissue maps based on the person image data I and include them in the model data M. Such maps can also be generated based on the person image data I, similar to the redness map R, by using a trained model obtained as a result of machine learning using data obtained by actually measuring with a measuring instrument as ground truth data. By using such subcutaneous tissue maps, the image processing device that uses this model data M can represent changes that appear in the appearance of a target object due to the influence of subcutaneous tissue, such as the oily shine (gloss) of the target object's skin.

[0041] Furthermore, regarding redness of the skin, individual redness maps R may be generated for each specific factor, such as changes in skin redness in response to blood flow, and used as needed. In this case as well, the redness maps R for each factor can be generated by performing machine learning using subcutaneous tissue maps obtained by measuring the structure of the subcutaneous tissue corresponding to each factor. An image processing device using this model data M can represent skin changes caused by various factors by selecting the subcutaneous tissue map to use according to the factors affecting the target object and mapping it to the surface of the target object together with the color map C.

[0042] The functions realized by the components described herein may be implemented by any circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, configured or programmed to realize the described functions. A processor is considered a circuit or processing circuitry, including transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

[0043] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If such hardware is a processor that is considered a type of circuitry, then such circuitry, means, or unit is a combination of hardware and software used to operate such hardware and / or processor.

[0044] This disclosure may include the following embodiments: [Item 1] An information processing device comprising a circuit configured to acquire image data of a person that is the basis of a target object, generate a color map which is a texture map representing the color of the appearance of the target object based on the acquired image data of the person, and generate a subcutaneous tissue map which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object based on the acquired image data of the person. [Item 2] An information processing device according to Item 1, wherein the circuit generates a redness map as the subcutaneous tissue map, representing the redness that occurs in the skin of a person due to the subcutaneous tissue. [Item 3] An information processing device according to Item 1 or 2, wherein the circuit generates the subcutaneous tissue map by performing multivariate analysis on the pixel values ​​included in an image based on the image data of the person. [Item 4] An information processing device according to Item 1 or 2, wherein the circuit generates the subcutaneous tissue map by inputting an image based on the image data of the person into a pre-trained model generated by machine learning. [Item 5] An information processing device as described in Item 4, wherein the trained model is a model generated by machine learning using image data obtained by imaging a real person and measurement data obtained by measuring the subcutaneous tissue of the person. [Item 6] An information processing method that acquires image data of a person that will be the basis of a target object, generates a color map which is a texture map representing the color of the appearance of the target object based on the acquired image data of the person, and generates a subcutaneous tissue map which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object based on the acquired image data of the person.[Item 7] Obtain image data of a person who is the source of the target object, generate a color map, which is a texture map representing the color of the appearance of the target object, based on the obtained image data of the person, and generate a subcutaneous tissue map, which is a texture map for determining the influence on the appearance of the target object by the subcutaneous tissue of the person, based on the obtained image data of the person, and store a program for causing a computer to execute the processing, a computer-readable non-temporary information storage medium.

[0045] 1 Information processing apparatus, 11 Control unit, 12 Storage unit, 13 Interface unit, 14 Display device, 15 Operation device, 21 Person image acquisition unit, 22 Color map generation unit, 23 Redness map generation unit, 24 Model data output unit.

Claims

1. An information processing device comprising: an image data acquisition means for acquiring image data of a person that will be the basis of a target object; a color map generation means for generating a color map, which is a texture map representing the color of the appearance of the target object, based on the acquired image data of the person; and a subcutaneous tissue map generation means for generating a subcutaneous tissue map, which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object, based on the acquired image data of the person.

2. An information processing device according to claim 1, wherein the subcutaneous tissue map generating means generates a redness map as the subcutaneous tissue map, which represents the redness that occurs in human skin due to the subcutaneous tissue.

3. An information processing apparatus according to claim 1 or 2, wherein the subcutaneous tissue map generation means performs multivariate analysis on pixel values ​​included in an image based on the human image data to generate the subcutaneous tissue map.

4. An information processing device according to claim 1 or 2, wherein the subcutaneous tissue map generation means inputs an image based on the person's image data into a pre-trained model generated by machine learning in advance, and generates the subcutaneous tissue map.

5. An information processing device according to claim 4, wherein the trained model is a model generated by machine learning using image data obtained by imaging a real person and measurement data obtained by measuring the subcutaneous tissue of the person.

6. An information processing method comprising: acquiring image data of a person that will be the basis for a target object; generating a color map, which is a texture map representing the color of the appearance of the target object, based on the acquired image data of the person; and generating a subcutaneous tissue map, which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object, based on the acquired image data of the person.

7. A program that causes a computer to perform the following steps: acquiring image data of a person that will be the basis for a target object; generating a color map, which is a texture map representing the color of the appearance of the target object, based on the acquired image data of the person; and generating a subcutaneous tissue map, which is a texture map for determining the influence of the subcutaneous tissue of the person on the appearance of the target object, based on the acquired image data of the person.

Images