Robotic system for applying a product and learning method

Abstract

A robotic system (1) for applying a product to a part of the human body, notably a face, comprising: - a robotic arm (10), - an applicator (11) mounted at the end of the arm, - at least one force sensor for measuring the force exerted by the applicator (11) during the application, - at least one optical sensor (13), - an arm control system, comprising: • a robot operating system (ROS) (100), and • an Al system (310) interacting in a closed loop with the robot operating system (100), this Al system having as inputs the outputs of the robot operating system and having been trained at least on a digital twin (320).

Description

Robotic system for applying a product and learning method

[0001] The present invention relates to systems for the automated application of a product to a surface, and more particularly but not exclusively to systems for the application of a makeup product to the face.Prior art

[0002] Automated application of a makeup composition can be useful to help people with a visual or motor disability apply makeup.

[0003] This can also make it possible to realize, in a reproducible manner, makeup which is elaborate or difficult to realize manually.

[0004] It can also be beneficial in being able to perform tests of application of a composition to an artificial surface such as a manikin head, in order to perform laboratory performance tests. These tests must be as representative as possible of real application conditions.

[0005] Robotic systems are known in which the arm bears an effector and causes it to follow a predefined trajectory in order to perform a task. Such systems require quite a considerable amount of preliminary work in programming the trajectory and are ill-suited to treating a moving surface, like the face of a person during a makeup session.

[0006] Devices which are held in one hand and are intended to be moved by a person on the face have also been proposed, comprising an inkjet printer head which is automatically controlled as it moves over the skin so as to apply a makeup composition according to a predefined pattern.

[0007] The design of a robotic system for realizing makeup must meet certain conditions.

[0008] First of all, in the case of makeup of a real person, the system must be able to realize the makeup despite the inevitable movements of the person’s face, and without exerting uncomfortable pressure on the person.

[0009] In addition, in the context of use at the point of sale or as an aid or development tool, the system must be capable of adapting rapidly to various facial morphologies and to the realization of various types of makeup.

[0010] Lastly, the makeup must be able to be realized relatively rapidly, safely for the user, and with a repeatability at least as good as if the makeup were realized by an experienced person.Disclosure of the invention

[0011] The aim of the invention is to propose a robotic system that allows makeup to be applied satisfactorily to a real face or test surfaces of various shapes.Summary of the invention

[0012] A subject of the invention is thus a robotic system for applying a product to a part of the human body, notably a face, comprising:a robotic arm movable along several degrees of freedom, notably six degrees of freedom,an applicator mounted at the end of the arm,at least one force sensor for measuring the force exerted, notably in a direction normal to the surface to be treated,by the applicator on the surface to be treated during the application,at least one optical sensor for imaging the surface to be treated,an arm control system, comprising:a robot operating system, connected to the robotic arm, to the force sensor and to the optical sensor, having as inputs actions for modifying the positioning of the arm, notably actions to be carried out on actuators of the robot, and as outputs the states of the arm, the force measured by the force sensor and the images from the optical sensor, andan AI system interacting in a closed loop with the operating system, this AI system having as inputs the outputs of the robot operating system and having been trained at least on a digital twin of at least a part of the human body, notably a face, and the robotic system with a deep reinforcement learning model, notably of the DQN type, in order to generate as outputs said actions for modifying the positioning of the arm.

[0013] Preferably, the deep reinforcement learning model is of the DQN type.

[0014] A Deep Q-Network (DQN) is a type of neural network used in deep reinforcement learning. It combines deep neural networks with a Q-learning algorithm to allow a machine to learn to make optimal decisions in a given environment. Q-learning is a reinforcement learning method that aims to learn an action-value function, called Q, that indicates the quality of an action in a given state. DQN learning models use neural networks to approximate the Q function, thereby allowing environments with a large number of possible states to be managed.

[0015] The robotic system according to the invention is capable of self-learning in a virtual environment, by virtue of the digital twin, and of generalizing what it has learned in the virtual environment in order to apply it to the real world.

[0016] The system is thus capable of autonomously applying a cosmetic composition to realize the makeup of a predefined region of the face, the application being carried out with a force that is sufficiently low not to cause discomfort but high enough nevertheless for the composition to be able to transfer onto the skin or lips.

[0017] The self-learning capability on a digital twin also facilitates the use of the robotic system on new supports and / or with new applicators or compositions.

[0018] Learning in a virtual environment again makes it possible to reduce the duration of human programming, because learning to realize a new makeup does not require complex manual programming of a new arm trajectory, and to gain in safety because learning does not generate any real movement of the arm.

[0019] The system is lastly capable of easily adapting to a new face and to the person’s movements during makeup.

[0020] Preferably, the optical sensor is borne by the arm. This improves the accuracy of image capture of the surface to be treated.

[0021] The optical sensor is preferably a camera generating depth information between the optical sensor and the surface to be treated. This depth information can be useful for controlling the distance between the application and the surface to be treated.

[0022] The system may be designed to apply a product to the face and comprise a position stabilization module in order to cause in real time the optical sensor, when borne by the arm, to position itself in a predefined manner with respect to the face in the frame of reference of the optical sensor, by acting on the control of the robot arm.

[0023] The robotic system may thus comprise a module for compensating for the movements of the person's face, notably aiming to follow the movement of the face so as to maintain it with a predefined, notably centered, position in the frame of reference of the camera, taking account of the movements of the arm, this servo-control being implemented in the control loop of the robot arm.

[0024] The robotic system may comprise a module for stabilizing the applied force, receiving a setpoint value from the AI system, and comprising a control loop, for example of the PID type, to act on the control of the arm in order to comply with this setpoint value as best as possible.

[0025] The system may comprise an artificial head defining the surface to be treated. This can allow tests of application of a cosmetic composition to this head to be carried out. The latter can also be used for the training of the robotic system.

[0026] The robotic system may comprise an interface allowing the user to select a zone to be treated and a product to be applied.

[0027] The system may comprise a mirror, the arm control system being designed to hold the mirror in front of the user, notably after realizing the makeup on their face, in order to allow them to appraise the result. The mirror may be designed to be grasped by the arm as a replacement for the applicator.

[0028] The product may be a makeup composition, notably selected from among lip and skin makeup products, notably from among lipstick, foundation and eyeliner.

[0029] A further subject of the invention is a learning method for a robotic system according to the invention, as defined above, wherein the AI system is trained using the digital twin of the surface to be treated and the robotic system.

[0030] The AI system may be trained first on the digital twin and then on an artificial head.

[0031] The training may be carried out in successive instances, generating the ST+1states of the next instance from the STstates of the current instance, so as to try and increase a reward RT.

[0032] This reward may be increased when at least one of the following conditions is met:completing the application over the entire zone to be treated,completing the application in less time,completing the application with a regular force,avoiding overflowing of the zone to be treated.

[0033] The STstates of the current instance may correspond to:the position of the arm bearing the applicator,the configuration of the surface to be treated,the value of the application force,the result of the current application.

[0034] The ST+1states of the next instance may be generated in response to actions generated by the system aiming to improve the reward, these actions possibly comprising the desired position of the end of the arm bearing the applicator in order to continue the application and the desired application force on the surface to be treated in order to continue the application.

[0035] The robot operating system is preferably designed to process the inputs from the digital twin in the same manner as those from the real environment. Thus, the digital twin may comprise a virtual robot arm having the same dimensions and the same control system as the real robot arm, the same optical sensor, a head mesh with at least one local physical property, and an application rendering module. The latter makes it possible to generate the resulting image of the virtual application of the makeup composition.Brief description of the drawings

[0036] The invention may be understood better from reading the following detailed description of a nonlimiting exemplary embodiment thereof, and from studying the appended drawing, in which:

[0037] schematically shows an example of a robotic system according to the invention,

[0038] shows the robotic arm on its own,

[0039] schematically illustrates the robotic arm control,

[0040] is an example of a graphical representation generated from the digital twin of the robotic system,

[0041] illustrates the location of points of reference that can be used to follow the face during the treatment of the surface to be treated,

[0042] is an example of a simulated view by the on-board optical sensor of a virtual face to be made up,

[0043] is an example of modeling a lip topography,

[0044] illustrates different steps of an example of a makeup method using the robotic system according to the invention, and

[0045] is an image showing an example of makeup of the virtual face.Detailed description

[0046] shows an example of a robotic system 1 according to the invention.

[0047] This system 1 comprises a robotic arm 10, shown on its own in, this arm being borne by a frame 15, which can house the electric power supply of the arm and its control interface.

[0048] The arm 10 is, for example, an arm with six degrees of freedom, known per se, comprising several motorized segments rotating, under the effect of motorized actuators, about respective axes of rotation.

[0049] The arm 10 can bear any type of gripper, for example a two-finger pincer 11, as illustrated.

[0050] The arm 10 incorporates a force sensor housed inside one or more elements 12 of the arm, which measures the force exerted in the axis of the pincer 11, on the latter, this axis preferably being coincident with the normal to the surface to be treated.

[0051] The arm 10 bears a color camera 13, which is preferably a camera generating depth information, that is to say that it not only delivers an image but also measures in real time the distance between the camera and the surface that is imaged. This camera may be borne by the last segment of the arm, as illustrated, in order to have a view in the axis of the pincer 11.

[0052] The system 1 may comprise a support 15a which makes it possible to pre-position on the frame 15 one or more applicators of makeup products, for example a lipstick 20 and an eyeliner 30 in the example illustrated. The frame 15 may also allow the pre-positioning of a mirror capable of being grasped by the arm.

[0053] An artificial head 40 may be positioned in front of the arm 10 by virtue of a tripod or any other suitable support.

[0054] The robotic system 1 comprises a computer system for controlling the arm which receives the information from the camera 13, from the force sensor 12 and from the arm 10. This computer system may comprise a microcomputer or any other computing means.

[0055] The software part that is executed by the computer system may be broken down into a part executed on a robot operating system (ROS) 100 environment and a part based on an artificial intelligence system 110, as illustrated in. This artificial intelligence system (also called "engine") may be based on the GYM software architecture from the company OPEN AI.

[0056] Communication between the ROS 100 environment and the artificial intelligence engine can be effected in real time using a UDP communication protocol.

[0057] The ROS 100 environment may have a software architecture called "sim-to-real", in order to manage two modes of operation, namely a simulated operation where the learning of the artificial intelligence engine can be carried out in virtual and a real operation where the artificial intelligence engine is used to control the arm in the real world. An example of a "sim-to-real" software architecture is described in the publication Zhao, W., Queralta, J.P. and Westerlund, T., 2020, December. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI) (pp. 737-744). IEEE.

[0058] The simulated operation is based on a digital twin of the robotic arm and its environment, as illustrated in.

[0059] This digital twin simulates the behavior of a robotic arm 10’ which is the replica of the arm 10. The virtual arm 10’ thus reproduces the dimensions of the real arm, its degrees of freedom, its actuation speeds, its constraints and physical collisions.

[0060] This arm 10’ thus bears a virtual pincer 11’ and a virtual camera 13’ which takes up the parameters of the real camera and the digital twin makes it possible to simulate the image that would be delivered by such a camera, which is similar to that delivered by the camera 13 borne by the arm 10, with the same field of view and observation point from the arm.

[0061] The digital twin also comprises a virtual applicator 20’ and one or more virtual faces 40’ and 50’.

[0062] These virtual faces 40’ or 50’ are realized for example with a mesh that reproduces the relief of a real face, as illustrated in, and this mesh can be associated with a texture model that simulates the appearance and at least one physical property of the surface such as local firmness. This texture makes it possible to generate force feedback information as a function of the position of the applicator application surface relative to the surface of the skin at rest. Thus, the force exerted in return on the applicator can be modeled as a function of the degree of pressure of the applicator on the skin.

[0063] The virtual face may be a model that is already available or be realized from a real face. An example of reconstructing a virtual face from a real face is described in the publication O'Toole, A.J., Price, T., Vetter, T., Bartlett, J. C. and Blanz, V., 1999. 3D shape and 2D surface textures of human faces: The role of “averages” in attractiveness and age. Image and Vision Computing, 18(1), pp. 9-19.

[0064] The simulation of the application of makeup, also called MVTO ("Makeup Virtual Try On"), can be effected by a technique known as deep graphic inversion, as described in the article Kips, Robin, et al. "Realtime Virtual Try On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers." Computer Graphics Forum. Vol. 41. No. 2. 2022.

[0065] The robotic system 1 is designed in such a way that regardless of the mode of operation of the arm, real or simulated, the states of the arm are generated in the same format and the control of the arm is carried out in the same manner.

[0066] The robotic system 1 is designed to control the arm in real time in order to realize the makeup of a given region of the face by virtue of learning carried out upstream with the digital twin.

[0067] The system 1 may use an artificial intelligence engine called DQN, the learning of which is deep reinforcement learning, in which the artificial intelligence engine receives a reward when the task is performed correctly. At each iteration, depending on the input states, the artificial intelligence engine tries to apply an action to increase the reward, and the next action is generated.

[0068] Reinforcement learning is a machine learning technique where an agent learns to act optimally from rewards received from the environment. The agent acts according to the state of the environment . The environment gives a reward to the agent depending on the effectiveness of the agent’s actions in completing the task.

[0069] In one example of implementation of the invention, the task of the agent consists in applying makeup to a target surface which may be a face, an application surface in the context of evaluating product performance, etc. The agent then is the robotic arm applying the makeup, and the environment the computer system with the camera and the application surface to which the product should be applied. The environment gives rewards depending on the completion state of the task.

[0070] The action model of the robotic arm, called policy, is a neural network a( | ) that, given the current state of the environment , predicts the trajectory that the robotic arm must travel to apply the makeup to the application surface, and also the values of the force sensor during the application .

[0071] The state of the environment is composed of the position of the robotic arm, the values of the force sensor, the current configuration of the application surface, and the targeted configuration of the application surface.

[0072] The predicted trajectory is represented by a set of consecutive points. In terms of neural network architecture, a( | ) is a deep neural network composed of three subnetworks. The first subnetwork takes as input the current configuration of the application surface, and the target configuration of the application surface, and produces a vector representation. The second subnetwork takes as input the position of the robotic arm, and also the values of the force sensor, and produces a vector representation. A third subnetwork combines the two vector representations to predict the trajectory that the robotic arm must follow, and also the values of the force sensor as the trajectory is traveled.

[0073] The reward is a signal sent back by the environment to the robot about its action. The reward is a scalar value that is all the higher if the configuration of the application surface is similar to the configuration of the targeted surface when the makeup is properly applied. Also, the reward is all the higher if the task is perfectly completed in a short period of time.

[0074] Agent learning consists in finding the right parameters, called weights in the context of a neural network. During the learning, the agent will apply the makeup, and step by step, improve its action model on the basis of the rewards received. The higher the rewards received, the better the policy. For the learning of the policy of the agent, an algorithm called an actor-critic algorithm can be used, as described in the publication Konda, Vijay, and John Tsitsiklis. "Actor-critic algorithms." Advances in neural information processing systems 12 (1999)., which consists in the introduction of an additional neural network called a value network which is an approximator for the value function consisting of the expectation of cumulative rewards at the end of the task, knowing that at steptthe environment was in the state . In terms of architecture, the value network can have the same structure as the neural network representing the policy of the agent, the only difference being that the value network predicts at the output the value of the state of the environment received at the input.

[0075] The network representing the policy of the agent and the network of the value function are jointly trained. Using the policy, the agent generates the actions, and the rewards received are used to update the network of the value function. Then the network of the value function is used to update the policy and promote the actions leading to states associated with strong value functions.

[0076] An example of an artificial intelligence engine suitable for implementing the invention is described in the publication Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. and Riedmiller, M., 2013, Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602.

[0077] The ROS 100 environment can generate the following states 300 as outputs at each iteration, as illustrated in:position states 301 of the arm (that is to say for example Cartesian coordinates of the pincer borne by the arm and orientation thereof,data 302 from the camera, namely an image from the camera with the associated depth data,force data 303 (that is to say the force value exerted on the applicator in response to its contact with the surface to be treated, notably the normal component of this force).

[0078] On the basis of the data 302 from the camera, the system can calculate the configuration of the application surface, and the current result of the application.

[0079] The artificial intelligence engine 310 processes both the states 311 from the simulation and those from the real world, the same data formats being used to describe the real or virtual states in the digital twin 320.

[0080] The ROS 100 environment also provides the AI engine 310 with the rewards 312 for accomplishing certain tasks.

[0081] For example, rewards are assigned for the successful completion of the following tasks:completing the application over the entire zone to be treated,completing the application in less time than previous times,completing the application with a regular force,avoiding overflowing of the zone to be treated.

[0082] The system 1 may comprise an interface for defining the type of makeup to be applied, and for selecting, where appropriate, the type of applicator or makeup composition.

[0083] The system 1 may be designed to perform pre-processing in order to recognize characteristic points on the image from the real camera and calculate the coordinates and the orientation of the face in the frame of reference of the arm.

[0084] The characteristic points of the face are, for example, those referenced 60 and illustrated in. These characteristic points are, for example, located at the outer corners of the eyes, at the tip of the nose, at the corner of the mouth and at the tip of the chin.

[0085] The system 1 is also designed to simulate the detection of the topology of the face on the basis of the data from the virtual camera 13’ of the digital twin.

[0086] An example of simulation of the image observed by the virtual camera 13’ is given in, for two viewing angles.

[0087] An example of determination of the topology of the lips of a face is illustrated in. This determination can be carried out in a prior step of recognition of the surface to be treated, real or virtual, by the robotic system.

[0088] The system 1 may comprise two stabilization modules 316 and 317, one for the position of the face and the other for the force exerted by the applicator.

[0089] The module 316 for stabilizing the force applied by the applicator is intended to apply a regular pressure during the application of the composition by means of the applicator. It may implement a control loop of the PID type.

[0090] At each instant T, the DQN model can calculate a desired force FTfor the continuation of the makeup and the force stabilization module 316 takes this value as a setpoint. A servo-control loop is set up to control the force during the application.

[0091] The module 317 for stabilizing the face, or visual servo-control, aims to follow the movement of the face, taking account of the movements of the arm. This servo-control is implemented in the control loop of the robot arm. An example of such visual servo-control is described in the publication Chaumette, F. and Hutchinson, S., 2006. Visual servo control. I. Basic approaches. IEEE Robotics & Automation Magazine, 13(4), pp. 82-90.

[0092] The training of the AI engine is carried out using the digital twin in successive instances, the model taking as inputs the states of the system at the instance T, and generating the actions to be carried out at the instance T+1 in order to maximize the reward.

[0093] The actions 314 are for example the target coordinates and orientations of the end of the arm bearing the applicator, according to a model with six degrees of freedom, and also the force to be applied.

[0094] In step 315, the ROS environment can convert this information into orientation values to be taken about each of the axes of articulation of the arm, using an inverse kinematic matrix, thus allowing precise control of the movements of the arm.

[0095] An example of makeup 41’ of a virtual face 40’ is illustrated in.

[0096] Once the training has been carried out on the digital twin, the system is ready for use in the real world.

[0097] An example of a makeup method using the robotic system according to the invention will now be described with reference to. Makeup using the robotic system 1 can be effected on an artificial head 40 as illustrated inor on a person.

[0098] In step 210, the manikin is placed in front of the robotic arm 10, ensuring that no obstacle impedes the movements of the arm.

[0099] In step 211, the camera 13 is used to recognize the characteristic points of the face and measure the distance to the face and guide the arm 10 toward the face. The system 1 keeps the face centered in the field of the camera 13.

[0100] In step 212, the system reconstructs a digital twin of the face by virtue of the images acquired by the camera 13. This digital twin includes the topology of the surface to be treated, for example the lips as illustrated in.

[0101] This digital twin of the face can be used in step 213 to allow a user to select, by virtue of the interface, a type of makeup, for example the application of a lipstick.

[0102] The makeup rendering is obtained by virtue of the MVTO module detailed above, and can be displayed on a screen with the digital twin, as illustrated in.

[0103] In step 214, the system executes a closed loop using the previously trained artificial intelligence engine, where the input is given by the states and the output by the actions that modify the states for the next iteration. This loop determines the trajectory of the arm and the force to be exerted during the application, the position of the arm being corrected in real time during the execution of this loop. Simultaneously, in step 215, the system follows the face so as to stabilize the view by the camera and regulates the applied force, by virtue of the aforementioned modules for stabilizing the position of the face in the frame of reference of the camera and the applied force.

[0104] Once the makeup has been realized, in step 216, the system can control the arm to rest the applicator.

[0105] When the made-up face is not that of a manikin but of a person, the system may be designed at this step to cause the arm to grasp a mirror in order to present it to the person who has just been made up.

[0106] Of course, the invention is not limited to the example which has just been described. The invention covers the application of any cosmetic, makeup or care, composition to human keratin materials.

[0107] Where appropriate, learning carried out using the digital twin can be supplemented by learning then carried out in the real world, for example on the manikin.

Claims

A robotic system (1) for applying a product to a part of the human body, notably a face, comprising:a robotic arm (10) having several degrees of freedom, notably six degrees of freedom,an applicator (11) mounted at the end of the arm,at least one force sensor for measuring the force exerted, notably in a direction normal to the surface to be treated, by the applicator (11) on the surface to be treated during the application,at least one optical sensor (13) for imaging the surface to be treated,an arm control system, comprising:a robot operating system (ROS) (100), connected to the robotic arm, to the force sensor and to the optical sensor, having as inputs actions for modifying the positioning of the arm and as outputs the states of the arm, the force measured by the force sensor and the images from the optical sensor, andan AI system (310) interacting in a closed loop with the robot operating system (100), this AI system having as inputs the outputs of the robot operating system and having been trained at least on a digital twin (320) of at least a part of the human body, notably a face, and the robotic system with a deep reinforcement learning model, notably of the DQN type, in order to generate as outputs (314) said actions for modifying the positioning of the arm.The system as claimed in claim 1, the optical sensor (13) being borne by the arm.The system as claimed in either of the preceding claims, the optical sensor (13) being a camera generating depth information between the optical sensor and the surface to be treated.The system as claimed in any one of the preceding claims, including a back-reference to claim 2, being designed to apply a product to the face and comprising a module for stabilizing the face, making it possible to compensate for movements of the face, and the position to cause the optical sensor to position itself in a predefined manner with respect to the face in the frame of reference of the optical sensor, by acting on the control of the robot arm.The system as claimed in any one of the preceding claims, comprising a module for stabilizing the applied force, receiving a setpoint value from the AI system, and comprising a control loop, notably of the PID type, to act on the control of the arm in order to comply with this setpoint value as best as possible.The system as claimed in any one of the preceding claims, comprising an artificial head (40) defining the surface to be treated.The system as claimed in any one of the preceding claims, the product being a makeup composition, notably selected from among lip and skin makeup products, notably lipstick, foundation and eyeliner.A learning method for a robotic system (1) as claimed in any one of the preceding claims, wherein the AI system is trained using a digital twin (320) of the surface to be treated and the robotic system.The method as claimed in claim 8, wherein the AI system trained on the digital twin (320) is then trained on an artificial head (40).The method as claimed in either of claims 8 and 9, the training being carried out in successive instances, generating the ST+1states of the next instance from the STstates of the current instance, so as to try and increase a reward RT.The method as claimed in claim 10, the reward being increased when at least one of the following conditions is met:completing the application over the entire zone to be treated,completing the application in less time,completing the application with a regular force,avoiding overflowing of the zone to be treated.The method as claimed in either of claims 10 and 11, the states STof the current instance corresponding to:the position of the arm bearing the applicator,the configuration of the surface to be treated,the force value,the result of the current application.The method as claimed in any of claims 10 to 12, the ST+1states of the next instance being generated in response to actions generated by the system aiming to improve the reward, the actions comprising the desired position of the end of the arm bearing the applicator in order to continue the application and the desired application force on the surface to be treated in order to continue the application.The method as claimed in any one of claims 8 to 13, the operating system being designed to process the inputs from the digital twin in the same manner as those from the real environment.The method as claimed in any one of claims 8 to 15, the digital twin comprising a virtual robot arm having the same dimensions and the same control system as the real robot arm, the same optical sensor, a head mesh with at least one local physical property, an application rendering module.

Images