Real-time 3D virtual clothing try-on and fabric simulation system on mobile devices

Abstract

This invention extracts body measurements with high accuracy by applying an artificial intelligence-based image processing process using two photographs (front and side) taken by users via a mobile device and the users' height information Subsequently, a digital twin of the user is created by applying the obtained measurements, in the form of proportional deformation by means of shape key (blend shape) technology, to a pre-baked 3D humanoid model available in Unity or a similar game / physics engine. The invention uses the shape key method again to convert garments from a single base model into different sizes and provides a real-time physics simulation. This simulation comprises advanced interactive features such as swaying of the fabric, a tension test, detection of tight and loose areas, and damage and abrasion / wear analyses. In addition, with augmented reality (AR) support, the model, which can be placed as a three-dimensional hologram in any environment desired by the user, operates in an optimised, real-time manner on mobile devices without requiring any dedicated rendering process. Accordingly, a realistic, secure and instant virtual clothing try-on capability is provided in the fields of e-commerce and virtual clothing.

Description

[0001] DESCRIPTION

[0002] REAL-TIME 3D VIRTUAL CLOTHING TRY-ON AND FABRIC SIMULATION SYSTEM ON MOBILE DEVICES

[0003] Technical Field

[0004] This invention relates to simulating, on a three-dimensional (3D) model, personal body measurements obtained from two (front and side) photographs of the human body taken using mobile devices by means of image processing and artificial intelligence techniques, and to performing real-time clothing try-on and cloth physics applications (real-time stretching, swaying, abrasion / wear, etc.) on this model. The invention is particularly suitable for use in e-commerce, augmented reality (AR)-enabled fashion applications, garment fit tests and virtual fitting environments.

[0005] State of the Art

[0006] Traditionally, it is necessary to try on a garment in order to determine whether it fits an individual. However, in online retailing, when users purchase garments without the opportunity to try them on, they encounter problems such as size mismatch and inadequate visualisation. As a result, return rates are high, and businesses incur losses due to logistics and inventory costs.

[0007] Some existing solutions include:

[0008] • Simple 2D photo-based virtual try-on applications: Body measurements or the physical behaviour of garments are not fully taken into account.• Professional 3D body scanners: These are expensive and are not accessible to everyone.

[0009] • Physics engine-based simulations: Although they are used more in the gaming and film industries, they are not common in real-time, online clothing try-on, or they may require high-end hardware.

[0010] The aspects that are lacking in the existing state of the art are as follows:

[0011] 1. In cases where sufficient depth measurements and optimum conditions are not available, image processing technologies are inadequate for obtaining body measurements accurately.

[0012] 2. Real-time simulations have not been sufficiently optimised for mobile devices in terms of rendering time and hardware requirements.

[0013] 3. Advanced fabric interactions, such as manual tension and damage tests, are not provided to the user.

[0014] OBJECT OF THE INVENTION

[0015] The objectives of this invention are as follows:

[0016] 1. User-friendly measurement: enabling the user to obtain highly accurate body measurements (arm length, leg length, waist / chest / leg circumference, etc.) by uploading two photographs (front and side) to the system using only a mobile device camera and by meeting simple clothing / visual conditions (white background, tight-fitting clothing), and by entering the user’s height.

[0017] 2. Real-time 3D model and clothing simulation: creating a digital twin of the user whose measurements have been obtained, and presenting garments simulated by a real-time physics engine with high performance even on a mobile device.

[0018] 3. AR and interactive features: placing the model in an augmented reality environment (by scanning the ground or the environment with the phone camera), enabling the fabric to be pulled and stretched, providingdamage / pilling simulation, and reporting tight and loose parts of the garment.

[0019] 4. Optimised architecture: ensuring real-time operation by managing model and garment diversity using shape keys (shape key) without requiring rendering time.

[0020] Accordingly, the invention aims to reduce return rates in the e-commerce sector, to evaluate prototypes rapidly in garment design and testing fields, and to improve AR-based fashion experiences.

[0021] Description of the Drawings

[0022] Figures 1 and 2: Schematic illustration of the mobile interface in which the user uploads front and side photographs.

[0023] Figure 3: General operation diagram of the invention.

[0024] Figure 4: Flow of the artificial intelligence module in which background removal, calibration and extraction of body measurements are performed.

[0025] Figure 5: Schematic representation of applying proportional deformation to the 3D human model by means of a shape key, and illustration of real-time virtual clothing try-on and physics-based simulation on a mobile device.

[0026] Figure 6: Illustration of the blend-shape and model creation and implementation process in a 3D modelling application.

[0027] Description of the Invention

[0028] This invention comprises an artificial intelligence algorithm that extracts body measurements from the user’s image captured by the camera, and a system that enables garments to be simulated accurately on the user by using these measurements.1. Photo Upload and Acquisition of Height Information

[0029] o The user uploads two photographs to the application running on the mobile device: a front photograph and a side photograph. The photographs must be taken in accordance with the conditions of a white background and tight-fitting clothing. In addition, the user’s height is provided to the application as an input.

[0030] 2. Background Removal and Contour Detection

[0031] o With the aid of artificial intelligence and image processing (for example, OpenCV, MediaPipe or similar libraries), the human silhouette in the photographs is detected.

[0032] o By means of a pre-trained model, the user is segmented from the background and contour lines are detected.

[0033] o By taking the height as a reference, the real-world equivalents of the pixel lengths in the user’s photograph are calculated.

[0034] 3. Extraction of Body Measurements

[0035] o Measurements such as the user’s arm, leg, waist and chest circumference are determined by marking human body keypoints using a machine learning algorithm trained in advance with the aid of a labelled photographic dataset.

[0036] o Lengths are obtained by combining the narrowest and widest intervals on the plane of the points indicating the locations of the body parts in the front and side photographs, by means of the two diameters thereby obtained.

[0037] o The obtained raw measurements are converted into precise centimetre measurements by using calibration coefficients determined in advance using the ratio of the user’s full pixel height and the real-world height, and by using these raw measurements in ellipse formulae designed in accordance with human body measurements.4. Creation of a 3D “Digital Twin”

[0038] o A pre-baked 3D humanoid model available in the system is adapted to the actual measurements by means of shape key (blend shape) technology.

[0039] o The user’s height, arm and leg lengths, and circumference measurements are converted into proportional deformation parameters on this standardised model. In this way, a user-specific 3D “digital twin” is generated.

[0040] 5. Garment Database and Simulation

[0041] o In the garment database, a single model and the corresponding shape key set are maintained for different sizes (S, M, L, XL, etc.).

[0042] o When the user selects the product in the basket, the application applies the relevant shape key parameters to the garment model and associates its physical size with the garment measurements.

[0043] o At the cloth simulation stage, methods enabling any garment physics manipulation within a physics engine (for example, Unity, etc.) are used (for example, Magica Cloth 2, which is a third-party cloth simulation library).

[0044] ■ By making dynamic additions to transform -based components (transform) and polygonal surface meshes (mesh), it simulates in real time the swaying of elements attached to the character (arms, trouser legs, etc.).

[0045] ■ As it uses a data-oriented architecture (data-oriented), it enables high performance; it can operate on the existing render pipeline without requiring a dedicated custom shader.

[0046] 6. AR and Interactive Features

[0047] o In AR mode, the user scans the ground with the phone camera and places a hologram-like projection of the digital twin at the desired position.o The user can move around it through 360 degrees using camera angles and examine the garment from different angles.

[0048] o By means of a manual fabric tension test, pulling / stretching is performed from a specific point of the garment and the fabric response (amount of stretching, recovery, etc.) is displayed in real time.

[0049] 7. Damage and Tight / Loose Simulation

[0050] o Algorithms that detect the compression points or loose areas of the fabric mark the parts experiencing the greatest contact or friction with the aid of a collision model.

[0051] o In specific activity scenarios, the system simulates the amount of abrasion / wear of the fabric or the level of damage.

[0052] 8. Real-time Rendering and Performance

[0053] o The entire process operates in an optimised manner on a mobile device; model diversity is managed by means of shape key deformations without requiring any additional rendering time.

[0054] o Owing to real-time operation, the user can try different garment options instantly.

[0055] 9. Model Sharing and Security

[0056] o The user’s body measurements and garment data are stored and / or shared securely in the cloud by means of compression and encryption methods.

[0057] o In this way, privacy and data security are also ensured.

[0058] 10. Screen Recording and Sharing

[0059] o The application allows the user to capture the real-time 3D simulation screen at any moment (for example, by taking a screenshot or recording a short video).o It provides an integration that enables the user to upload the screenshot / simulation recording directly to social media platforms or other sharing channels.

[0060] o During sharing, only the visual recording relating to the simulation is transferred in a manner that protects the privacy of user data; the user’s actual personal measurements or other sensitive data are not shared.

[0061] Industrial Applicability

[0062] This invention can be implemented particularly in online clothing stores, virtual fitting rooms and the fashion industry in order to improve users’ shopping experiences. It is also highly possible for it to be used by personal style consultants and fashion designers.

Claims

CLAIMS1. A method for extracting body measurements using two photographs, namely a front photograph and a side photograph, taken by a user with a mobile device and using the user’s height information, and subsequently performing real-time three-dimensional virtual clothing try-on, characterised in that it comprises:• removing the background in the photographs by means of artificial intelligence-based image processing and determining the user’s body contour, • calculating a pixel-to-centimetre conversion factor proportional to the user’s height information,• obtaining body measurements such as arm / leg lengths, waist / chest / leg circumference, etc. by means of landmark correspondences in the front and side photographs,• creating a “digital twin” by applying proportional deformation on a pre-baked 3D humanoid model by means of shape key / blend shape technology using these measurements, and• performing real-time simulation of the garment by means of a physics engine or a cloth simulation library.

2. The method according to claim 1, further comprising a step of converting said garments from a single base model into different sizes by means of a shape key.

3. The method according to claim 1 or 2, characterised in that, in an augmented reality (AR) mode, the user scans the environment with the mobile device camera to place a hologram of the 3D model and is provided with 360-degree interaction capability.

4. The method according to any one of claims 1 to 3, characterised in that it comprises, during cloth simulation, a manual tension test in which a specific point of the fabric is pulled and stretched such that stretching, friction and relaxation responses of the fabric are displayed in real time.

5. The method according to any one of claims 1 to 4, characterised in that it comprises damage or abrasion / wear simulation of the garment and is capable of calculating, over time, the amount of wear of the regions that undergo the greatest interaction by using collision data.

6. The method according to any one of claims 1 to 5, characterised in that said simulation operates in real time on a mobile device and is carried out by means of shape key deformations without requiring an additional rendering process.

7. The method according to any one of claims 1 to 6, characterised in that the 3D model and cloth simulation data used are subjected to compression and encryption steps prior to sharing or cloud synchronisation, thereby ensuring data security.

8. The method according to any one of claims 1 to 7, characterised in that the system comprises performing garment simulation by using Python -based image processing (for example, OpenCV and similar libraries), artificial intelligence techniques and a physics engine.

Images