Information processing device and information processing method

Abstract

A server 10 includes: a generation means 12 that generates intermediate images in which clothing has been put onto each of a plurality of human body models from clothing data for the clothing and 3D data acquired from a human body database at which 3D data for the human body models has been recorded, a writing means 13 that writes the generated intermediate images into an intermediate database, an extraction means 14 that extracts an intermediate image that uses a human body model that corresponds to the body shape of a target user from the intermediate images recorded at the intermediate database in accordance with body shape information for the target user, a synthesis means 15 that synthesizes individual elements for the target user into the extracted intermediate image to obtain a synthesized image, and an output means 17 that outputs the synthesized image.

Description

Information Processing Apparatus and Information Processing Method

[0001] The present invention relates to virtual fitting using a human body model. In particular, it relates to a technique for highly accurately and rapidly generating a virtual fitting image adapted to a user's body shape by using a human body model image prepared in advance.

[0002] Conventionally, virtual fitting systems using two-dimensional image generation AI have been known. With this system, it is difficult to accommodate various factors such as the user's body shape, pose, and hairstyle, and the generated image may be unnatural or distorted.

[0003] Patent Document 1 discloses a technique for transmitting data for outputting a 3D model composed of specific elements among 3D modeling data managed by a server to an information processing apparatus serving as a terminal.

[0004] Japanese Patent No. 6799883

[0005] In conventional virtual fitting using two-dimensional image generation AI, there have been problems such as unnaturalness of the generation result and long calculation processing time.

[0006] According to one aspect of the present invention, there is provided: a generation means for generating, for each of a plurality of human body models, an intermediate image in which the clothes are worn on the human body model, from 3D data acquired from a human body database storing 3D data of the plurality of human body models and clothes data of the clothes; a writing means for writing the generated intermediate image into an intermediate database; an extraction means for extracting an intermediate image using a human body model corresponding to the target user's body shape from among the intermediate images stored in the intermediate database in accordance with body shape information of the target user; a combining means for combining a unique element of the target user with the extracted intermediate image to obtain a combined image; and a first output means for outputting the combined image. The present invention provides an information processing apparatus having the above.

[0007] The information processing apparatus may further include a second output means that outputs data for causing a confirmation terminal to display the combined image generated by the generation means for human check.

[0008] The body shape information may include scan data obtained by 3D scanning the target user.

[0009] This information processing device may have an instruction means for instructing the target user to assume the pose that the human body model is taking in the intermediate image recorded in the intermediate database for the purpose of 3D scanning.

[0010] The aforementioned unique element may include the face of the target user.

[0011] The aforementioned unique element may include the hairstyle of the target user.

[0012] The synthesis means may synthesize the unique element into the composite image by swapping the unique element with a corresponding element in the intermediate image.

[0013] The synthesis means may also synthesize the unique elements into the synthesized image by drawing the unique elements into the intermediate image using generating AI.

[0014] This information processing device may have a finishing means for performing a finishing process on the composite image, and the first output means may output the composite image after the finishing process has been performed.

[0015] This information processing device may have additional means for adding a new human body model in which the body shape of a human body model recorded in the human body database has been modified according to the target clothing.

[0016] This information processing device has a identifying means for identifying the range to which the clothing is intended in a space defined by a plurality of parameters indicating the body shape of a human body model, and the additional means may add new human body models such that the number or density of human body models belonging to the range satisfies a condition.

[0017] This information processing device may have a presentation means for displaying a composite image in which another human body model from the human body database, which fits the garment and is similar in body shape to the target user, is dressed in the garment, if the body shape information of the designated user that matches the target user does not fit the garment.

[0018] The clothing data may include information regarding how the clothing is worn, and the generation means may generate the intermediate image corresponding to how the clothing is worn.

[0019] The clothing data may include information about the material of the clothing, and the generation means may generate the intermediate image corresponding to the material.

[0020] Another aspect of this disclosure provides an information processing method comprising the steps of: generating a composite image of each of the multiple human body models wearing the clothing, from 3D data of multiple human body models obtained from a human body database which records 3D data of multiple human body models and clothing data of clothing; writing the generated composite image to an intermediate database; extracting a composite image from the composite images recorded in the intermediate database that uses a human body model corresponding to the body shape of the target user, according to the body shape information of the target user; compositing the target user's unique elements onto the extracted composite image; and outputting the composite image with the unique elements composited.

[0021] According to the present invention, virtual try-on images tailored to the user's body shape can be generated with high accuracy and speed.

[0022] A diagram showing an overview of the virtual fitting system 1. A diagram showing the functional configuration of the virtual fitting system 1. A diagram showing the hardware configuration of the server 10. A diagram showing the hardware configuration of the user terminal 20. A diagram showing the hardware configuration of the scanner 30. A diagram showing an overview of the processing of the virtual fitting system 1. A sequence chart showing the intermediate image generation process. A diagram illustrating the clothing database 113. A diagram illustrating the human body database 112. A diagram illustrating the distribution of human body models. A diagram illustrating the target range. A diagram illustrating the expansion of human body models. A sequence chart showing the intermediate image confirmation and registration process. A diagram illustrating the confirmation screen. A diagram illustrating the intermediate database 111. A sequence chart showing the composite image generation process. A diagram illustrating the user database 114. A diagram illustrating the instruction screen. A diagram illustrating the fitting screen. A diagram illustrating the suggestion screen.

[0023] 1. Diagram 1 shows an overview of the virtual fitting system 1. The virtual fitting system 1 is an information processing system that provides a virtual fitting service to users. Virtual fitting refers to simulating what it would look like to wear clothes on a computer. In the virtual fitting system 1, the results of the virtual fitting are provided as a 2D image. The virtual fitting system 1 includes a server 10, a user terminal 20, a scanner 30, and a confirmation terminal 40.

[0024] Server 10 is a computer device that stores and manages various types of data. For example, Server 10 stores human body data and clothing data. Human body data is 3D data of a human body model. Clothing data is data related to clothing, such as the shape, pattern, and material of the clothing. Clothing data may be 3D or 2D data. Server 10 uses the human body data, clothing data, and user-specific elements to provide a composite image of the user virtually trying on clothes.

[0025] The user terminal 20 is a computer device used by users who perform virtual try-on. The user terminal 20 can be, for example, a smartphone, a tablet, or a personal computer. The user terminal 20 requests various data from the server 10 and displays various data received from the server 10 in response to user operations.

[0026] The scanner 30 is a device that measures the user's body shape. The scanner 30 includes, for example, a camera, a depth sensor, a rangefinder, etc. The scanner 30 measures the user's three-dimensional shape and outputs 3D scan data. The 3D scan data is data representing the user's body shape and includes, for example, the dimensions of each part of the user.

[0027] The verification terminal 40 is a computer device used by workers to check intermediate images. Intermediate images refer to images of a human body model dressed in clothing. The verification terminal 40 is, for example, a personal computer. The verification terminal 40 displays intermediate images and accepts operations from the worker.

[0028] Figure 2 shows the functional configuration of the virtual fitting system 1. The virtual fitting system 1 includes a server 10, a user terminal 20, and a scanner 30.

[0029] Server 10 includes storage means 11, generation means 12, writing means 13, extraction means 14, synthesis means 15, output means 16, output means 17, instruction means 18, and control means 19. Storage means 11 stores various data and programs. It includes an intermediate database (intermediate DB) 111, a human body database (human body DB) 112, a clothing database (clothing DB) 113, and a user database (user DB) 114. The human body database 112 is a database that records 3D data of multiple human body models. The clothing database 113 is a database that records clothing data (e.g., 3D data). Generation means 12 generates intermediate images. An intermediate image is an image (in this example, a 2D image) of a human body recorded in the human body database 112 wearing clothes recorded in the clothing database 113. Writing means 13 writes the intermediate images generated by generation means 12 to the intermediate database 111. The intermediate database 111 is a database that records intermediate images. Server 10 generates intermediate images in advance for various combinations of human body models and clothing and stores them in the intermediate database 111. User database 114 is a database that stores data such as the user's body shape and facial image. Output means 16 outputs the intermediate images generated by generation means 12 to confirmation terminal 40 for confirmation (an example of a second output means).

[0030] Extraction means 14 extracts intermediate images using a human body model corresponding to the target user's body type from the intermediate images recorded in the intermediate database 111. Instruction means 18 instructs the target user to assume the pose taken by the human body model in the intermediate images recorded in the intermediate database 111 for 3D scanning. Synthesis means 15 synthesizes the target user's unique elements with the extracted intermediate images to obtain a composite image. Output means 17 outputs the composite image generated by the synthesis means 15 (an example of a first output means). Control means 19 controls the entire virtual fitting system 1.

[0031] Server 10 further includes a finishing means 191, an adding means 192, a display means 193, and a identifying means 194. The finishing means 191 performs finishing processing on the composite image. The adding means 192 adds a new human body model to the human body database 112, which has a modified body shape from a human body model recorded in the human body database 112, according to the target clothing. The identifying means 194 identifies the range targeted by the target clothing in a space defined by a plurality of parameters indicating the body shape of the human body model.

[0032] The user terminal 20 has an acquisition means 21 and a display means 22. The acquisition means 21 acquires user data. The user data is data about the user, and includes, for example, the user's body shape data and the user's facial image. The display means 22 displays various information.

[0033] The scanner 30 has an acquisition means 31 and a transmission means 32. The acquisition means 31 acquires at least a portion of the user data. The transmission means 32 transmits the acquired user data to the server 10.

[0034] Figure 3 shows the hardware configuration of server 10. Server 10 is a computer device having a CPU 101, memory 102, storage 103, and communication IF 104. The CPU 101 is a processing device that performs various processes according to a program. The memory 102 is a main memory that functions as a work area when the CPU 101 executes a program. The storage 103 is a non-volatile auxiliary storage device that stores various data and programs. The storage 103 stores an intermediate database 111, a human body database 112, a clothing database 113, and a user database 114. The communication IF 104 is a device that communicates with other information processing devices according to a predetermined communication standard (e.g., Ethernet®).

[0035] In this example, the program stored in the storage 103 includes a program (hereinafter referred to as the "server program") that causes the computer to function as a server in the virtual fitting system 1. When the CPU 101 is executing the server program, the CPU 101 is an example of a generation means 12, a writing means 13, an extraction means 14, a synthesis means 15, an output means 16, an output means 17, an instruction means 18, a control means 19, a finishing means 191, an addition means 192, a presentation means 193, and a specific means 194, and at least one of the memory 102 and the storage 103 is an example of a storage means 11.

[0036] Figure 4 shows the hardware configuration of the user terminal 20. The user terminal 20 is a computer device having a CPU 201, memory 202, storage 203, communication IF 204, input device 205, and output device 206. The user terminal 20 is, for example, a smartphone, tablet terminal, or personal computer. The CPU 201 is a processing device that performs various processes according to a program. The memory 202 is a main memory that functions as a work area when the CPU 201 executes a program. The storage 203 is a non-volatile auxiliary storage device that stores various data and programs. The communication IF 204 is a device that communicates with other information processing devices via a network according to a predetermined communication standard (for example, Ethernet®). The input device 205 is a device that receives input from the user. The input device 205 includes, for example, a touch screen or a microphone. The output device 206 is a device that outputs information. The output device 206 includes, for example, a display or a speaker.

[0037] Figure 5 shows the hardware configuration of the scanner 30. The scanner 30 is a computer device having a CPU 301, memory 302, storage 303, communication IF 304, input device 305, output device 306, and sensor group 307. The CPU 301 is a processing device that performs various processes according to a program. The memory 302 is a main memory that functions as a work area when the CPU 301 executes a program. The storage 303 is a non-volatile auxiliary storage device that stores various data and programs. The communication IF 304 is a device that communicates with other information processing devices according to a predetermined communication standard (e.g., Ethernet®). The input device 305 is a device that receives input from the user. The output device 306 is a device that outputs information. The sensor group 307 measures the body shape of the user, who is the subject. In one example, the sensor group 307 includes multiple sets of cameras and distance sensors. A 3D model of the subject is obtained from the image obtained from the camera and the distance map obtained from the distance sensor. The user's 3D model is an example of body shape information that shows the user's body shape. In addition to or instead of images and distance, the sensor group 307 may measure other body shape information, such as weight or body fat percentage.

[0038] The scanner 30 is a device that scans, or measures, the user who is the subject of whole-body data, and has a scanning room (or booth) (the scanning room is not shown). The scanning room is a space in which the user performs measurements to generate whole-body data, and in one example it has a frame and wall materials. The sensor group 307 is installed in the scanning room.

[0039] The CPU 301 controls the sensor group 307 and collects measurement data from the sensor group 307. The CPU 301 transmits the collected measurement data to the server 10. The server 10 generates whole-body data (i.e., 3D data) from the measurement data, or records the measurement data as whole-body data.

[0040] 2. Operation diagram 6 is a diagram showing an overview of the processing of the virtual fitting system 1. The virtual fitting system 1 broadly performs four processes: intermediate image generation processing, intermediate image confirmation and registration processing, composite image generation processing, and final result output processing.

[0041] The intermediate image generation process generates intermediate images based on 3D data of a human body model obtained from the human body database 112 and clothing data obtained from the clothing database 113. When a target garment 401 is selected from the clothing database 113, the target garment 401 is tried on by a human body model selected from the human body database 112 (402). In many cases, multiple human body models are selected for a single target garment 401, and multiple intermediate image candidates are generated. These intermediate image candidates are recorded in the initial database 403.

[0042] The intermediate image verification and registration process involves verifying the intermediate images generated in the intermediate image generation process (404), and registering the verified intermediate images in the intermediate database 111. In this example, the verification of the intermediate images is performed by a human. Human verification ensures the quality of the intermediate images. The intermediate images registered in the intermediate database 111 are used in the composite image generation process. Intermediate images that are judged to be NG in verification (404) are sent back to the fitting (402). At this time, the instructions or parameters related to the fitting are modified, and a retry is performed. Up to this point, the system performs these processes spontaneously, regardless of the end user's instructions. The virtual fitting system 1 generates intermediate images, for example, during times when the system's processing or load is low.

[0043] The subsequent processing is performed when a target user issues a specific instruction for virtual fitting. The composite image generation processing is processing for extracting, from among intermediate images recorded in the intermediate database 111, an intermediate image that uses a human body model corresponding to the body shape of the target user. First, the virtual fitting system 1 acquires body shape information 405 of the target user. The virtual fitting system 1 selects, from among the intermediate images recorded in the intermediate database 111, an intermediate image using a human body model having a body shape close to the body shape information 405 (406). The virtual fitting system 1 further acquires a unique element 408 of the target user. The user's unique element is, for example, the user's face or hairstyle. The virtual fitting system 1 replaces, in the intermediate image, an element corresponding to the unique element of the target user with the unique element 408 (409). A fitting image 410 is obtained through the replacement. The virtual fitting system 1 performs finishing processing 411 on the fitting image 410. A final result 412 is obtained through the finishing processing.

[0044] The final result output processing is processing for outputting the final result 412 to the user terminal 20. The final result 412 includes an image (simulating a state) of the target user wearing the specified clothing.

[0045] FIG. 7 is a sequence chart showing intermediate image generation processing in the virtual fitting system 1. Details of the processing outlined in FIG. 6 will be described below. The intermediate image generation processing is processing performed by the server 10, and is processing for generating an intermediate image based on 3D data of a human body model recorded in a human body database 112 and clothing data recorded in a clothing database 113.

[0046] In step S701, the server 10 identifies the target garment. In one example, the process in Figure 7 is initiated when a new garment is registered in the garment database 113. In this case, the newly registered garment is the target garment. Alternatively, the data of a garment already registered in the garment database 113 may be updated, in which case the updated garment may be identified as the target garment. Furthermore, the target garment may be identified by explicit instructions from the administrator of the virtual fitting system 1 or a specific user (e.g., an apparel manufacturer).

[0047] Figure 8 illustrates a clothing database 113. The clothing database 113 is a database that stores information about clothing. The clothing database 113 contains multiple records. Each record contains a clothing ID, attributes, size information, and clothing data. The clothing ID is an identifier for identifying clothing. Attributes are information that indicates the attributes of clothing, and include detailed sub-items, such as brand, category, size range, color, and price. The brand indicates the brand or business that provides the clothing. The category indicates the category of the clothing, for example, classification by shape (tops, bottoms, outerwear, or dresses, etc.). The size range indicates the size range of the clothing, for example, S, M, L, etc. The color indicates the color of the clothing. The price indicates the price of the clothing. Attributes may also include other information, such as material or use. In this example, clothing of the same design but different sizes are given the same clothing ID, but different clothing IDs may be assigned to each size.

[0048] The size information indicates the size of the human body model corresponding to the garment. The size information indicates the size corresponding to the garment at a specific part of the human body in each size development. In the example of the figure, the parts include neck circumference, chest circumference, arm length, waist circumference, and hip. For each of these parts, the size corresponding to the garment is recorded. The corresponding size is, for example, a numerical value specified by the garment provider (for example, in cm or inch units), and indicates the optimal size estimated to provide comfortable wearing when the garment is worn, or the extreme size that physically allows wearing the garment. This size may be represented by a single numerical value (recommended value, maximum value, or minimum value), or may be represented by a numerical range (minimum value to maximum value). Whether the size information is defined by the optimal size or the extreme size may be uniformly determined as an operation rule in the virtual fitting system 1, or may be defined for each garment provider. In addition, it is not necessary to describe a size numerical value for all parts listed as items in the garment database 113. If there is no restriction on, for example, neck circumference depending on the design of the garment, and the garment can be worn by a person of any size, the neck circumference may be left blank (or a null value).

[0049] Garment data is data of a garment. In one example, the garment data is an image (2D image) of the garment. In another example, the garment data is a 3D model (3D data) of the garment. In this example, information indicating a location where garment data is stored is recorded in the garment database 113. Alternatively, the garment data itself may be recorded in the garment database 113.

[0050] In step S702, the server 10 specifies requirements for the human body model used for generating the intermediate image. As described with reference to FIG. 8, size information is defined for the target garment. Therefore, a human body model on which the target garment can be virtually fitted should have size requirements (conditions or restrictions). The server 10 refers to the garment database 113 and specifies the requirements that the human body model for virtually fitting the target garment should satisfy.

[0051] In step S703, the server 10 queries the clothing database 113. This query includes information that specifies the size requirements for the human body models. The clothing database 113 searches the database in response to the query from the server 10 and returns information to the server 10 regarding human body models that meet the requirements (step S704). This return includes, for example, the number of human body models that meet the requirements.

[0052] Upon receiving a response from the human body database 112, the server 10 determines whether the number of human body models meets a condition. This condition is, for example, that the number of human body models is greater than or equal to a predetermined lower limit. The lower limit may be a common setting value used throughout the virtual fitting system 1, or it may be defined for each clothing provider, or it may be defined or specified for each user. If the server 10 determines that the number of human body models meets the condition, it proceeds to step S708. If the server 10 determines that the number of human body models does not meet the condition, it proceeds to step S705.

[0053] In step S705, the server 10 expands the number of stored human body models. Expanding the number of human body models means generating new human body models by modifying the 3D data of human body models already recorded in the human body database 112. Specifically, expanding human body models includes processes such as increasing or decreasing the size of existing human body models. The algorithm for modifying existing human body models is defined in the server 10. For example, the server 10 generates a new human body model by enlarging or shrinking the entire existing human body model by a certain percentage. Alternatively, the server 10 may generate a new human body model by enlarging or shrinking only specific parts of the existing human body model. Enlarging a part includes making that part fatter or increasing its length. Shrinking a part includes making that part thinner or decreasing its length. The expansion of the number of human body models will be explained in more detail below.

[0054] Figure 9 illustrates a human body database 112. The human body database 112 is a database that stores 3D data of a human body model. A human body model is a three-dimensional model that mimics the shape of a human body. The human body database 112 contains multiple records. Each record contains data about one human body model. Each record contains a model ID, size information, and 3D data. The model ID is an identifier used to identify the human body model. The size information is information that indicates the size of each part of the human body model, such as height, weight, neck circumference, chest circumference, upper arm circumference, arm length, waist circumference, hip circumference, inseam, and thigh circumference. The size of each item is indicated by a specific numerical value (in cm or inches). The 3D data is data of a 3D model of the human body. In this example, the human body database 112 records information indicating the location where the 3D model data is stored. Alternatively, the 3D data itself may be recorded in the human body database 112.

[0055] In this example, the human body data recorded in the human body database 112 is classified into two types: original data and augmented data. Original data is 3D data obtained by scanning real people with a 3D scanner. For example, scanning 100 people will yield 100 sets of 3D data. Augmented data is 3D data obtained by modifying the original data. As mentioned above, various modifications are possible for a single human body model, so applying 100 different modifications to one set of original data will yield 100 sets of augmented data.

[0056] In creating the human body database 112, the human model is scanned (i.e., photographed) while wearing thin, plain clothing that fits the body well and can be easily combined with other clothing later.

[0057] Figure 10A illustrates the size distribution of a human body model. For simplicity, in Figure 10, we consider only two parameters related to the size of the human body model: height and weight. We consider a two-dimensional space with height on the horizontal axis and weight on the vertical axis. To consider the sizes of n parts in the human body model, we can extend this to an n-dimensional space. The black circles in Figure 10A represent the original human body model stored in the human body database 112.

[0058] As shown in Figure 10A, the human body database 112 stores human body models of various heights and weights. Specifically, the human body database 112 stores human body models of various body types, such as short, tall, thin, and obese.

[0059] Figure 10B is an example of the target range. Target range At indicates the range of body types that correspond to the target garment (i.e., those who can wear the target garment). The position or size of the target range varies depending on the target garment. This is because the target users differ depending on the garment, specifically depending on the design, brand, or intended use. For example, sportswear is made to be very tight, so the target range is narrow. On the other hand, pajamas are made to be loose, so the target range is wide.

[0060] The human body models included in the target range A are the human body models that can be used when generating intermediate images using the target clothing. In the example in Figure 10B, there are two human body models included in the target range A. Whether this is a large or small number depends on the target customer group of the target clothing, but in the extreme case where there are no human body models included in the target range A, intermediate images cannot be generated. If there are a sufficient number of human body models in the target range A, intermediate images can be generated without any problems. Thus, if the number of human body models included in the target range A is less than the threshold, the number of human body models is expanded. If the number of human body models included in the target range A is greater than the threshold, the number of human body models is not expanded. The threshold may be set commonly in the virtual fitting system 1, or it may be set in smaller units, such as for each garment.

[0061] Figure 10C illustrates the expansion of the human body model. Figure 10C illustrates the expansion of the human body model in relation to the target range At shown in Figure 10B. In Figure 10C, the white circles indicate the expanded human body model. For example, a minimum number of human body models that should exist within the target range At is defined, and the human body model is expanded to satisfy this minimum number. In the example of Figure 10C, 18 human body models are added, bringing the total number of human body models within the target range At to 20.

[0062] When expanding the number of human body models, server 10 retrieves data for the human body models to be expanded (two original data in the example of Figure 10C) from the human body database 112. Server 10 then modifies these human body models to obtain the expanded human body models.

[0063] Refer to Figure 7 again. In step S706, the server 10 is instructed to write the newly generated human body model data to the human body database 112. In step S707, the human body database 112 adds a record of the new human body model.

[0064] In step S708, the server 10 requests data for a human body model that meets the size requirements for the target garment from the human body database 112. The human body database 112 outputs the requested human body model data to the server 10 (step S709). In step S710, the server 10 requests data for the target garment from the garment database 113. The garment database 113 outputs the requested garment data to the server 10 (step S711).

[0065] In step S712, the server 10 processes the clothing data of the target garment and the human body data acquired in step S709 to generate an intermediate image. In one example, the server 10 generates an intermediate image using an external system's generation AI (not shown). This generation AI generates an image of a person wearing a garment from a 2D image of the person and a 2D image of the garment. In this example, since the human body data is 3D data, the server 10 generates a 2D image of the human body from this 3D data. In one example, the pose and viewpoint position of the human body for the intermediate image are determined in the virtual fitting system 1. The server 10 has the human body model assume a predetermined pose and generates a 2D image of the human body model taken from a viewpoint at a predetermined position. Alternatively, a recommended pose and recommended viewpoint position for the human body model may be defined for each garment, and the server 10 may have the human body model assume a recommended pose and generate a 2D image of the human body model taken from a viewpoint at the recommended position. Server 10 inputs instructions to its generation AI to generate an image of a human body model dressed in clothing. These instructions include a 2D image of the human body model and a 2D image of the clothing.

[0066] The generation AI generates a 2D image of the human body model wearing the specified clothing, following instructions from the server 10. The generation AI outputs the generated 2D image to the server 10. The server 10 records the 2D image obtained from the generation AI in the initial database 403. This 2D image is a candidate for an intermediate image.

[0067] Figure 11 is a sequence chart showing the intermediate image verification and registration process in the virtual fitting system 1. The process in Figure 11 is initiated by a predetermined event. The triggering event is, for example, the receipt of explicit instructions from the verifier, or the arrival of a predetermined time.

[0068] In step S1301, the server 10 sends a confirmation request to the confirmation terminal 40. The confirmation request includes candidate intermediate images to be confirmed. Candidate intermediate images are selected from the initial database 403. In step S1302, the confirmation terminal 40 displays the candidate intermediate images received from the server 10 and requests confirmation from the verifier (S1302).

[0069] Figure 12 illustrates a confirmation screen 1400. The confirmation screen 1400 provides a user interface for presenting an intermediate image to the user and inquiring whether or not to adopt the intermediate image. The confirmation screen 1400 includes a region 1401, a button 1402, and a button 1403. Region 1401 is the region for displaying the generated intermediate image. The intermediate image is an image of a human body model dressed in clothing, and by reviewing this intermediate image, the user can check the size and design of the clothing.

[0070] Button 1402 is selected by the user when they want to use the intermediate image displayed in the intermediate image display area 1401. When button 1402 is pressed by the user, the intermediate image is registered in the intermediate database 111.

[0071] Button 1403 is selected by the user when they do not want to use the intermediate image displayed in the intermediate image display area 1401. When button 1403 is pressed by the user, the intermediate image is not registered in the intermediate database 111.

[0072] This screen allows the user of the verification terminal 40 to check the displayed intermediate image. Since these candidate intermediate images are generated by a generation AI, depending on the accuracy of the generation AI, there is a possibility that images of unsuitable quality for presentation to the end user may be generated. Therefore, the virtual fitting system 1 employs a process of verification by human eyes.

[0073] The verification terminal 40 transmits the verification result of the intermediate image to the server 10 (S1303). The verification result is, for example, information indicating whether or not the intermediate image can be adopted. Based on the verification result received from the verification terminal 40, the server 10 registers the intermediate image in the intermediate database 111 (S1304). That is, if the verification result indicates adoption, the server 10 registers the intermediate image in the intermediate database 111. In addition, regardless of the verification result, the server 10 requests the initial database 403 to delete the intermediate image. The initial database 403 deletes the requested intermediate image. As a result, intermediate images that have been verified are deleted from the initial database 403.

[0074] If button 1403 is pressed (i.e., the displayed intermediate image is not selected), server 10 may perform a process to cause the generating AI to regenerate the intermediate image. The process to regenerate the intermediate image includes a process to have the reviewer input improvements to the image. Improvements to the image are entered, for example, as text. That is, server 10 presents a UI object to the reviewer to input improvements to the image. The reviewer inputs improvements to the image from this UI object. Server 10 inputs instructions to the generating AI, including these improvements, a 2D image of the human body model, and a 2D image of the clothing. Candidate intermediate images regenerated by these instructions are recorded in the initial database 403.

[0075] Through the above process, the new intermediate image is stored in the intermediate database 111 and becomes available for use in subsequent processing.

[0076] Figure 13 shows an example of the data structure of the intermediate database 111. The intermediate database 111 includes an intermediate image ID, information about the human body, information about clothing, and image data. The intermediate image ID is identification information for identifying the intermediate image. The information about the human body includes a model ID. The information about clothing includes a clothing ID. The image data is data of the intermediate image in which the human body model is dressed in clothing.

[0077] The intermediate database 111 is used to store the intermediate images generated during the virtual fitting process of this system. The intermediate images are images of a human body model virtually dressed in various clothes, and these image data are stored in the intermediate database 111 and referenced during subsequent user fitting simulations. For example, by searching the intermediate image of a human body model that is closest to the user's body type in the intermediate database 111 and compositing the user's unique elements such as their face onto that image, a more natural and real-time virtual fitting can be achieved.

[0078] The intermediate images stored in the intermediate database 111 are not simply a collection of generated image data, but also include information about the body shape data of the human body model and the combination of clothing. This allows the system to quickly select an appropriate intermediate image based on the user's body shape information. Furthermore, the intermediate images are registered in the database after undergoing quality checks by humans as needed, thereby ensuring the quality and reliability of the virtual try-on.

[0079] Figure 14 is a sequence chart showing the process of generating a composite image in the virtual fitting system 1. The process in Figure 14 is initiated, for example, when a user requests a virtual fitting from the user terminal 20.

[0080] In step S1501, the user terminal 20 sends a fitting request to the server 10. The fitting request includes the garment ID of the garment to be tried on and the user ID of the user who will be trying on the garment (hereinafter referred to as the "target user"). When the server 10 receives the fitting request from the user terminal 20, it reads the target user's body shape data from the user database 114 (S1502).

[0081] Figure 15 illustrates a user database 114. The user database 114 is a database that stores information about users. The user database 114 contains multiple records. Each record stores information about one user. The user ID is an identifier used to identify a user. Each record includes attributes, size information, and 3D data.

[0082] Attributes are information that indicates the user's attributes, and in this example, include information such as name, date of birth, gender, and race. Body shape data includes information on the size of various parts of the user's body, such as height, weight, neck circumference, chest circumference, waist circumference, hips, upper arm circumference, arm length, thigh circumference, and inseam. 3D data is data for the user's 3D model.

[0083] Refer to Figure 14 again. In step S1503, the server 10 identifies a human body model to be used for fitting based on the target user's body shape data. Specifically, the server 10 requests the human body database 112 to search for a human body model similar to the target user. The human body database 112 responds to this request and searches for a human body model that has the body shape most similar to the target user's body shape data. The human body database 112 notifies the server 10 of the model ID of the human body model found in the search.

[0084] Server 10 sends a request for an intermediate image to the intermediate database 111 (S1504). This request includes the model ID identified in step S1503 and the clothing ID specified in step S1501. The intermediate database 111 reads the intermediate image corresponding to the requested model ID and clothing ID and sends it to Server 10 (S1505). Server 10 retrieves this intermediate image from the intermediate database 111.

[0085] In step S1506, the server 10 transmits an instruction to take a full-body image of the user. This instruction includes information that identifies the pose the human model is taking in the intermediate image. The information that identifies the pose is, for example, an image of the human model or a (pre-assigned) pose ID. The user terminal 20 displays a screen that prompts the user to take a full-body image in accordance with the instruction from the server 10. This screen includes an instruction that the user assumes the same pose as the human model in the intermediate image.

[0086] Figure 16 illustrates an instruction screen displayed in step S1506. The instruction screen is used to instruct the user to assume the same pose as a human body model when performing a 3D scan with the scanner 30. The instruction screen includes an image 1601 showing this pose. Image 1601 is, for example, an image of a human body model recorded in the human body database 112. The instruction screen includes a message 1602, for example, "Please assume the same pose as this model." This allows the user to take a photograph in the same pose as a human body model registered in the intermediate database 111. The user's own image, taken with the camera of the user terminal 20 (for example, the so-called front camera), may be displayed side by side or overlaid on this screen. When the user presses the capture button 1603, the user terminal 20 takes a picture of the user (step S1507).

[0087] In this way, by displaying an instruction screen, the user can easily assume the appropriate pose. In this example, although a full-body image of the user is taken in step S1506, only a part of the user (for example, the face) is used to generate the composite image, not the entire body. In this example, by having the user take a full-body image, the user can be given the feeling that their own image is being used for the virtual try-on. In other words, it leads to an improved user experience.

[0088] Refer to Figure 14 again. In step S1508, the user terminal 20 transmits the captured user image to the server 10 (S1508). In step S1509, the server 10 obtains the user's unique elements from the received user image (S1509). In this example, the image of the user's shoulders and above is extracted as the user's unique elements. That is, these unique elements include the user's shoulders, neck, face, and hair. In step S1510, the server 10 synthesizes the user's unique elements into the acquired intermediate image. This synthesis process replaces the face and hairstyle of the human body model in the intermediate image with those of the user. In one example, this synthesis is a process of swapping (or replacing) the user's unique elements with the elements corresponding to these unique elements (i.e., the human body model's shoulders, neck, face, and hair) in the intermediate image. In another example, this synthesis is a process in which the generating AI draws these unique elements into the intermediate image. Thus, a synthesized image is obtained.

[0089] Server 10 performs finishing processing on the composite image (S1511). This finishing processing includes, for example, background compositing and lighting adjustments. Server 10 then sends the finished composite image to the user terminal 20 as a fitting image (i.e., the final result) (S1512). The user terminal 20 displays the fitting image.

[0090] Figure 17 illustrates a virtual try-on screen displayed on the user terminal 20. The virtual try-on screen provides an interface for the user to check the results of the virtual try-on. The virtual try-on screen includes a region 1701, which is the area for displaying the try-on image. This allows the user to feel as if they are actually trying on clothes. In reality, the part of the try-on image from the shoulders down is not the user's but a human body model, but the body type is selected to be as close as possible to the user's. Generally, the process of compositing clothing images onto human images takes time. That is, if the process of compositing the image of the target clothing onto the user's full-body image is performed in real time, the user will experience waiting time. In contrast, in this embodiment, the time-consuming process is performed in advance using images of other people (human body models) and recorded in the intermediate database 111. When a request for virtual try-on is received from the user terminal 20, the server 10 reads an intermediate image using a human body model that is closest to the user's body type from the intermediate database 111 and presents it to the user, thus providing the user with a virtual try-on image in a shorter time.

[0091] If no intermediate image matching the user's body shape data exists in the intermediate database 111 (which often means the user's body shape does not match the target of the clothing), the server 10 outputs an error message to the user terminal 20. In this case, the server 10 may, in lieu of or in addition to the error message, offer suggestions to the user regarding their body shape.

[0092] Figure 18 illustrates a suggestion screen displayed on the user terminal 20. This suggestion screen provides an interface for making additional suggestions to the user based on the results of the virtual try-on.

[0093] The suggestion screen displays a message 1801, for example, "If you shape up by -3cm in chest, -3cm in waist, and -2cm in upper arms, you can wear this outfit." This suggests that if the virtual try-on results indicate that the selected clothing is slightly too small for the user's body type, improving their physique will allow them to wear the clothing more attractively. This message includes quantitative suggestions on which body parts should be increased or decreased and by how much. This suggestion is obtained by comparing the user's body data with the human body model that most closely matches the user's body type among the human body models used in the intermediate image using the target clothing. This suggestion may also be personalized based on information such as the user's interests, preferences, and past purchase history.

[0094] 3. Modifications The present invention is not limited to the above-described embodiment, and various modifications are possible. Several modifications are described below. Two or more of the following modifications may be used in combination.

[0095] 3-1. Expanding the Number of Human Body Models The triggers and specific methods for expanding the number of human body models are not limited to the examples of embodiments. For example, the expansion of the number of human body models may be performed independently of the addition of new clothing to the clothing database 113. In one example, the server 10 records the usage history of intermediate images. Here, the server 10 identifies human body models that are frequently used as intermediate images (i.e., human body models that are frequently used by users with similar body types). The server 10 expands the number of human body models based on these human body models.

[0096] 3-2. Generation of Intermediate Images The method for generating intermediate images is not limited to the examples of the embodiments. In the examples of the embodiments, the data of the target clothing was provided as a 2D image, but the data of the target clothing may also be provided as a 3D model. Here, instead of generating the intermediate images by the generation AI, a simulation (i.e., calculation) may be performed in which a 3D model of clothing is put on a 3D model of a human body, the human body model wearing this target clothing is made to assume a predetermined pose, and a 2D image is obtained taken from a predetermined viewpoint.

[0097] Furthermore, although the embodiment shows an example where the intermediate image is a still image, the intermediate image is not limited to a still image. The intermediate image may be a video. This allows the user to see the human body model wearing the clothes moving, providing a more realistic try-on experience.

[0098] Furthermore, when generating intermediate images, the material of the clothing may be input as a parameter to the intermediate image generation means (such as generation AI). Even clothes made from the same pattern may wrinkle differently depending on the material, such as polyester, cotton, or silk. If the intermediate image generation means can reflect the difference in material, the material may also be input as a parameter.

[0099] 3-3. Confirmation of Intermediate Image Candidates The process of confirming intermediate image candidates is not limited to the examples of the embodiments. In the examples of the embodiments, a verifier, i.e., a human, confirmed the intermediate image candidates, but the decision of whether or not to adopt an intermediate image candidate may be automatically made by AI or the like. Alternatively, the confirmation of intermediate image candidates may be omitted, and all generated intermediate images may be recorded in the intermediate database 111.

[0100] 3-4. User Body Shape Information The method for obtaining user body shape information is not limited to the examples of the embodiments. In the embodiments, user body shape information was read from the user database 114, but user body shape information may also be obtained by taking a photograph with the user terminal 20 or scanner 30.

[0101] 3-5. User-Specific Elements In this embodiment, an example was shown in which an image from the shoulders up is used as the user's unique element, but the unique element is not limited to this. For example, at least one of the following may be used as a unique element: face, hairstyle, body type, skin color, and tattoos. Alternatively, a combination of multiple elements may be used as a unique element. This allows the user to experience virtual try-on with an avatar that is closer to themselves, thereby increasing the realism of the try-on images.

[0102] 3-6. How to Wear Clothes When a human body model wears a garment, the way it is worn (or styled) is not limited to just one. For example, a long-sleeved shirt or jacket can be worn with the sleeves rolled up, in addition to the usual way of wearing it with the sleeves extended. Or, a jacket or blouson with a zipper closure can be worn with the zipper open or closed. A collared garment can be worn with the collar flat or with the collar up. In one example, variations in how to wear clothes are defined in the clothing database 113. For example, an item called "How to Wear" is set as an attribute for each garment, and further sub-items such as "Sleeves Rolled Up," "Front Zipper," and "Collar Up" are set as details. For garments where "Sleeves Rolled Up" is "Possible," when generating intermediate images, intermediate images with rolled-up sleeves and intermediate images without rolled-up sleeves are generated. When obtaining a composite image from the intermediate images, it is specified whether or not to roll up the sleeves. This may be specified by the user, or it may be automatically set by server 10 or the like based on user attributes or current trends.

[0103] Regarding how clothes are worn, further variations can be considered when combining them with other clothing. For example, when combining a top such as a shirt with bottoms such as pants, variations arise in whether the shirt hem is tucked into the bottoms or left out (so-called tuck-in or tuck-out). In one example, variations in how clothes are worn when combined are defined in the clothing database 113. For example, an item called "combination" is set as an attribute for each garment, and further sub-items "combination partner" and "how to wear" are set as details. The value of the item "combination partner" specifies the type of clothing, such as "pants," and the value of the item "how to wear" specifies the way to wear it, such as "tuck-in" or "tuck-out." These ways of wearing may also be added as attributes in the intermediate image.

[0104] 3-7. When selecting a combination of clothing and a human body model, the human body model that fits the clothing is not limited to a human body model within the size range specified (by the clothing provider) for the clothing. For example, as part of fashion, it is possible to intentionally wear oversized clothing in a baggy manner, or undersized clothing in a tight manner. The virtual fitting system 1 may generate intermediate images for such combinations of clothing and human body models that do not fit. When obtaining a composite image from the intermediate images, the use of such mismatched size combinations may be specified by the user, or it may be automatically set by the server 10 or the like based on user attributes or current trends.

[0105] 3-8. Types of Human Body Models In the human body database 112, human body models may be classified into categories. Categories may be classified, for example, by body fat percentage, i.e., from muscular to fatty. Even if the sizes of two human body models at a reference position (neck circumference, chest circumference, waist circumference, thigh circumference, etc.) are almost the same, the way they look when wearing clothes may differ depending on whether the human body model is muscular or fatty. To address such cases, the virtual fitting system 1 may increase the parameters in the human body database 112 to enable more detailed matching with the user's body type.

[0106] 3-9. Various UI Screens The UI screens are not limited to the examples of the embodiments. For example, the display screen for virtual try-on images may be a multi-view. A multi-view means displaying images of a human body model from multiple different viewpoints on a single screen. A multi-view may include, for example, images from the front, back, and side. In one example, the generation AI generates an intermediate image of the multi-view, and the intermediate image of the multi-view is recorded in the intermediate database 111. The intermediate image of the multi-view is selected as the intermediate image used for virtual try-on, and user-specific elements are embedded in the intermediate image of the multi-view. Alternatively, the intermediate image itself may be a single view (for example, only a front view). In this case, the server 10 may input the try-on image with the user-specific elements embedded into the generation AI that generates a multi-view from a single view, and obtain a multi-view try-on image.

[0107] The display screen for the virtual try-on images may be in free view mode. Free view means displaying the try-on images from any viewpoint specified by the user on the user terminal 20 viewing the virtual try-on images. In one example, the try-on images provided by the server 10 are in single view mode. In the viewer installed on the user terminal 20, free view try-on images are generated from single view try-on images. This viewer either has built-in AI to generate free view try-on images from single view try-on images, or accesses external AI to generate free view try-on images from single view try-on images.

[0108] The display screen for the virtual try-on images may be a video. In one example, the try-on images provided by the server 10 are 2D still images. A viewer installed on the user terminal 20 generates a video from the 2D still images. This viewer either has built-in AI to generate videos from 2D still images, or accesses external AI to generate videos from 2D still images.

[0109] Here, the content described as being generated on server 10 may be generated on the viewer of user terminal 20, and the content described as being generated on the viewer of user terminal 20 may be described on server 10. Furthermore, this content may be generated on hardware other than server 10 and user terminal 20. In addition, combinations of generating a multi-view from a single view, generating a free view from a multi-view, etc., may be used.

[0110] 3-10. Items to be tried on In the embodiments described above, clothing was shown as an example of an item to be tried on, but the items to be tried on are not limited to clothing. For example, the virtual try-on system 1 may be used to try on other items such as hats, glasses, accessories, shoes, or bags. This expands the scope of application of the virtual try-on system 1 and makes it usable in various fields.

[0111] 3-11. System Applications One possible application of the virtual fitting system 1 is online shopping, but its applications are not limited to this. For example, the virtual fitting system 1 may be used for other purposes such as games and social networking services. This expands the usage scenarios of the virtual fitting system 1 and makes it possible to provide new customer experiences.

[0112] 3-12. In other modified embodiments, some of the functions described as being provided by the virtual fitting system 1 may be implemented in a system other than the virtual fitting system 1. Also, some of the functions described as being implemented in the server 10 in the embodiments may be implemented in a device other than the server 10. In particular, various databases may be implemented in a device other than the server 10, in which case the server 10 accesses the database to obtain the necessary information.

[0113] The configuration of the virtual fitting system 1 is not limited to the examples of the embodiments. Multiple physical devices may cooperate to function as devices such as the server 10 of the virtual fitting system 1. In particular, the server 10 may be a physical server or a virtual server (including so-called cloud).

[0114] Some of the functional elements described in Figure 2 may be omitted, or new functions may be added. Also, the flow and sequence described in the embodiment are merely examples, and some steps may be rearranged, omitted, or new steps may be added. Furthermore, the UI described in the embodiment is merely an example, and any UI may be adopted as long as it can present or obtain the required information. Also, the database described in the embodiment is merely an example, and the data structure or data items may be anything.

[0115] The program executed by the CPU 101 or other processor may be provided in a state where it is recorded on a computer-readable non-temporary recording medium (e.g., CD-ROM), or it may be provided in a state where it can be downloaded on a server.

[0116] 1...Virtual fitting system, 10...Server, 11...Storage means, 12...Generation means, 13...Writing means, 14...Extraction means, 15...Synthesis means, 16...Output means, 17...Output means, 18...Instruction means, 19...Control means, 20...User terminal, 21...Acquisition means, 22...Display means, 30...Scanner, 31...Acquisition means, 32...Transmission means, 40...Confirmation terminal, 101...CPU, 102...Memory, 103...Storage, 104...Communication IF, 111...Intermediate database, 112...Human body database, 113...Clothing database, 114...User database, 191...Finishing means, 192...Addition means, 193...Presentation means, 194...Special 201...CPU, 202...Memory, 203...Storage, 204...Communication IF, 205...Input device, 206...Output device, 301...CPU, 302...Memory, 303...Storage, 304...Communication IF, 305...Input device, 306...Output device, 307...Sensor group, 401...Target clothing, 403...Initial database, 405...Body shape information, 408...Unique elements, 410...Try-on image, 411...Finishing process, 412...Final result, 1400...Confirmation screen, 1401...Area, 1402...Button, 1403...Button, 1601...Image, 1602...Message, 1603...Shoot button, 1701...Area, 1801...Message

Claims

1. An information processing device comprising: a generation means for generating an intermediate image of each of the multiple human body models wearing the clothing, obtained from 3D data of multiple human body models and clothing data of the clothing, obtained from a human body database which records 3D data of multiple human body models; a writing means for writing the generated intermediate image to the intermediate database; an extraction means for extracting an intermediate image from the intermediate images recorded in the intermediate database that uses a human body model corresponding to the body shape of the target user, according to the body shape information of the target user; a synthesis means for obtaining a composite image by combining the unique elements of the target user with the extracted intermediate image; and a first output means for outputting the composite image.

2. The information processing apparatus according to claim 1, further comprising a second output means for outputting data to be displayed on a verification terminal for human review of the composite image generated by the generation means.

3. The information processing apparatus according to claim 1, wherein the body shape information includes scan data obtained by 3D scanning the target user.

4. The information processing apparatus according to claim 3, further comprising an instruction means for instructing the target user to assume the pose that the human body model is taking in the intermediate image recorded in the intermediate database for the purpose of 3D scanning.

5. The information processing apparatus according to claim 1, wherein the unique element includes the face of the target user.

6. The information processing apparatus according to claim 1, wherein the unique element includes the hairstyle of the target user.

7. The information processing apparatus according to claim 1, wherein the synthesis means synthesizes the unique element into the synthesized image by swapping the unique element with a corresponding element in the intermediate image.

8. The information processing apparatus according to claim 1, wherein the synthesis means synthesizes the unique elements into the synthesized image by drawing the unique elements into the intermediate image using generating AI.

9. The information processing apparatus according to claim 1, further comprising a finishing means for performing a finishing process on the composite image, wherein the first output means outputs the composite image after the finishing process has been performed.

10. The information processing apparatus according to claim 1, further comprising additional means for adding a new human body model in which the body shape of a human body model recorded in the human body database has been modified according to the target clothing.

11. Information processing apparatus according to claim 10, comprising: a identifying means for identifying the range to which the clothing is intended in a space defined by a plurality of parameters indicating the body shape of a human body model; and the additional means for adding new human body models such that the number or density of human body models belonging to the range satisfies a condition.

12. The information processing apparatus according to claim 1, further comprising a presentation means for presenting a composite image in which another human body model from the human body database, which fits the garment and is similar in body shape to the target user, is dressed in the garment, when the body shape information of a designated user that is suitable for the target user does not fit the garment.

13. The information processing apparatus according to claim 1, wherein the clothing data includes information regarding how the clothing is worn, and the generating means generates the intermediate image according to how the clothing is worn.

14. The information processing apparatus according to claim 1, wherein the clothing data includes information relating to the material of the clothing, and the generating means generates the intermediate image corresponding to the material.

15. An information processing method comprising the steps of: generating a composite image of each of the multiple human body models wearing the clothing, from 3D data of multiple human body models obtained from a human body database containing 3D data of multiple human body models and clothing data of the clothing; writing the generated composite image to an intermediate database; extracting a composite image from the composite images recorded in the intermediate database that uses a human body model corresponding to the body type of the target user, according to the body type information of the target user; compositing the target user's unique elements onto the extracted composite image; and outputting the composite image with the unique elements composited.

Images