Method, apparatus and product for determining effect pigment data as sociated with an effect pigment; client-server architecture and use

Abstract

A computer-implemented method (10) for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, the method comprising receiving (12) coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels; segmenting (14), using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment; generating (16) at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on classification data sets comprising labelled image data of a plurality of different effect pigments; and determining (18), based on the at least one effect pigment class, the effect pigment data associated with the effect pigment; corresponding computer program elements, apparatuses, client-server architectures, and uses.

Description

[0001] 231417WO01 - Secondary Filing Text

[0002] BASF Coatings GmbH

[0003] 1

[0004] METHOD, APPARATUS AND PRODUCT FOR DETERMINING EFFECT PIGMENT DATA ASSOCIATED WITH AN EFFECT PIGMENT; CLIENT-SERVER ARCHITECTURE AND USE

[0005] TECHNICAL FIELD

[0006] The invention relates to identifying effect pigments in coatings, and more particularly to determining effect pigment data associated with the effect pigments comprised in the coatings.

[0007] TECHNICAL BACKGROUND

[0008] The present disclosure relates, in general terms, to identification of effect pigments in coatings.

[0009] To protect and I or decorate a substrate, e. g. a body of a vehicle such as a car, it may be provided with a surface coating. Surface coatings may be applied as a monocoat, two-layer coating comprising a colorcoat as basecoat and a clearcoat for protecting the colorcoat from damage, or a three-layer coating (tricoat) further comprising a mid-coat layer for creating a more aesthetically pleasing finish, for example.

[0010] Formulations for coatings (coating formulations) may be pigmented coating formulations, e. g. opaque coating formulations, comprising one or more pigments. Types of pigments comprise color pigments and effect pigments, for example. The color pigments have an optical effect that is based on selective light absorption and I or light scattering. The effect pigments have an optical effect that is based on directed light reflection, light refraction and I or light interference. A pigmented coating formulation comprising only a color pigment or color pigments may be referred to as solid color formulation. A pigmented coating formulation comprising an effect pigment or effect pigments may be referred to as effect coating formulation. For example, a coating on a car body is prepared by applying at least one coating formulation to a surface of the car body and curing the applied coating formulation(s). The curing may be performed per applied coating formulation, or jointly for the applied coating formulations, such that the applied coating formulations are cured simultaneously. Typical coatings comprise at least one colorcoat layer having a pigmented coating formulation, and at least one clearcoat layer on top of the colorcoat layer. An effect pigment of a given type comprises effect pigment flakes made of a given material and typically having non- uniform contours, i. e. irregular edges, non-uniform sizes, that may cause different size distributions for different effect pigments. Materials may comprise aluminum and mica, for example.

[0011] Effect pigments flakes in coatings may overlap with each other and I or may create agglomerates. Moreover, effect pigments interacting with solid pigments may potentially lead to color variations of the effect pigment flakes and / or affect the texture appearance. Thus, identification of effect 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0012] 2 pigments, that uses image analysis, for a unique classification of the effect pigments is extremely difficult and complex.

[0013] For repairing an existing coating, that may have been damaged in a car collision, by stone chipping or by scratches, for example, an unknown automotive color needs to be identified, so that a matching coating can be applied. Identification typically starts with identifying the effect pigments used for the coating to be matched. The step of identifying the effect pigments is still a demanding, mostly manual step requiring comprehensive expert knowledge. A colorist examines a color sample under a microscope and identifies candidates of effect pigment types for a first color match. The colorist may need to test repeatedly a number of coating compositions, even when the vehicle’s original equipment manufacturing (OEM) coating is known. For a set of new vehicles having a given OEM coating, their color and I or appearance may vary owing to a number of reasons including differences, for example, in application techniques and atmospheric conditions, that may vary from production plant to plant, time of year and year to year, for example. Further, the car body and add-on parts such as bumpers and casings of exterior rearview mirror may be made of different material, have different textures, have been produced in different production plants and I or have been coated in different paint shops. Thus, coatings of the car body and add-on parts, even for a supposedly same color, may have different formulations, requiring different repair formulations in case of repair. Moreover, colors and I or appearances of the vehicles may change individually owing for a number of reasons including use, car, environmental conditions and solar radiation, for example.

[0014] Thus, color adjustment of a sample coating formulation to match the color and I or appearance of a reference coating is an iterative process. As such, starting with a preliminary sample coating formulation, a sample coating formulation usually has to be adjusted several times until an adjusted sample coating having a color and I or appearance that sufficiently match the color of the reference coating has been found, rendering the color adjustment process time consuming and expensive.

[0015] For developing new coating formulations and I or for analyzing coating formulations of unknown coatings, tools, such as digital tools, may support developers and colorists alike.

[0016] However, there remains a need for identifying an effect pigment in a coating with high accuracy. Further, there remains a need for identifying an effect pigment with a low concentration I occurrence in a coating. Furthermore, there remains a need for identifying an effect pigment with rare presence / use in coatings. 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0017] 3

[0018] SUMMARY

[0019] In an aspect, the disclosure relates to a computer-implemented method for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, the method comprising receiving coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels; segmenting, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment; generating at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on classification data sets comprising labelled image data of a plurality of different effect pigments; and determining, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

[0020] The method may provide for identifying the effect pigment in the coating with a high accuracy. Thus, the method may provide for a more reliable determination of the effect pigments being responsible for a visual appearance of the coating. Further, the method may provide for a significantly improved color-matching process, even without requiring extensive knowledge on effect pigments. Furthermore, the method may reduce a number of test I trials I iterations required to produce a sample coating that visually matches appearance to a reference coating associated with the image data being analyzed. Moreover, the method may reduce an amount of energy; materials comprising, for example, raw material, coating material, and consumable; and / or waste may be reduced. In case of providing the segmenting and the classifying separately, the method may also allow implementing, testing, training and updating the steps individually. Thus, the method may reduce time for implementing and I or maintaining the method and I or increase efficiency. As a result, time for, cost for and I environmental impact of developing new coating formulations and I or for analyzing coating formulations of unknown coatings, for example when repairing an existing coating, may be reduced. Additionally or alternatively, the method may provide separate changes of the segmenting and I or the classifying, and, thus, flexible adjustment to use of new pigments in coatings. Thus, the method may provide for identifying the new effect pigments in the coatings. 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0021] 4

[0022] In another aspect, the disclosure relates to a computer-implemented method for training a data- driven classification model for generating at least one effect pigment class by classifying segmented image data, the method comprising applying classification data sets comprising labelled image data of a plurality of different effect pigments in at least one coating to the data-driven classification model; wherein the labelled image data are respectively associated with at least one of effect pigment type, effect pigment category, effect pigment property, effect pigment size and effect pigment color of the different effect pigments.

[0023] The method may provide for correlating the segmented image data and the labelled image data. The method may provide for improved identification of the effect pigment in the coating. Additionally or alternatively, the method may provide a change of the classification separate a change of the segmentation, and, thus, flexible adjustment to use of new pigments in coatings. Thus, the method may provide for less complex, more data- efficient and I or more robust training and I or shorter training times. Moreover, the method may allow to implement the data-driven classification model using, for example, an artificial neural network (ANN), a convolutional neural network (CNN), deep learning, an EfficientNet such as a EfficientNet BO, a residual neural network (residual network, ResNet) such as a ResNet 50, machine learning (ML) and I or ensemble learning.

[0024] In another aspect, the disclosure relates to a computer program element with instructions, which when executed on at least one computing node is configured to carry out the steps of the methods.

[0025] The computer program element may also provide for identifying the effect pigment in the coating with a high accuracy.

[0026] In another aspect, the disclosure relates to an apparatus for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, the apparatus comprising an input device configured to receive coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels; at least one computing node configured to: segment, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment; generate at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0027] 5 classification model has been trained on data sets comprising labelled image data of a plurality of different effect pigments; and determine, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

[0028] The apparatus may provide for identifying the effect pigment in the coating with a high accuracy. Thus, the apparatus may provide for a more reliable determination of the effect pigments being responsible for a visual appearance of the coating. Further, the apparatus may provide for a significantly improved color-matching process, even without requiring extensive knowledge on effect pigments. Furthermore, the apparatus may reduce a number of test I trials I iterations required to produce a sample coating that visually matches appearance to a reference coating associated with the image data being analyzed. Moreover, the apparatus may reduce an amount of energy; materials comprising, for example, raw material, coating material, and consumable; and I or waste may be reduced. In case of providing the configuration to segment and configuration to classify separately, the apparatus may also allow implementing, testing, training and updating the configurations individually. Thus, the apparatus may reduce time and I or necessary amount of training data for implementing and I or maintaining the apparatus and I or increase efficiency. As a result, time for, cost for and I environmental impact of developing new coating formulations and I or for analyzing coating formulations of unknown coatings, for example when repairing an existing coating, may be reduced. Additionally or alternatively, the apparatus may provide separate changes of the segmenting and I or the classifying, and, thus, flexible adjustment to use of new pigments in coatings. Thus, the apparatus may provide for identifying the new effect pigments in the coatings.

[0029] In another aspect, the disclosure relates to an apparatus for providing coating data associated with a coating on a surface area of an object, the apparatus comprising an output device configured to provide the coating data associated with the coating to an apparatus for determining effect pigment data associated with an effect pigment comprised in the coating, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels.

[0030] The apparatus allows to provide the apparatus for determining the effect pigment data associated with the effect pigment comprised in the coating on the surface area of the object with the coating data associated with the coating data. Thus, the apparatus may provide for improved identification of the effect pigment in the coating with a high accuracy. Additionally or alternatively, the apparatus provides for a central and I or more efficient use of the apparatus for determining the effect pigment data, for example in a remote configuration. 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0031] 6

[0032] In another aspect, the disclosure relates to a client-server architecture for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, wherein: a client is configured to provide coating data associated with the coating, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels; and a server is configured to: receive the coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels segment, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment; generate at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on data sets comprising labelled image data of a plurality of different effect pigments; and determine, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

[0033] The client-server architecture allows to provide the coating data from the client to the server. Thus, the client-server architecture may provide for improved identification of the effect pigment in the coating with a high accuracy. Additionally or alternatively, the client-server architecture provides for a central and I or more efficient determination of the effect pigment data. Thus, the client-server architecture allows centrally to provide the server with resources such configuration to segment, a configuration to classify and an up-to-date database comprising a plurality of formulations for coatings, for example. The server may centrally determine the effect pigment data for a plurality of clients. The client-server architecture may provide for a cloud-based solution. The client-server architecture may provide for a cloud-based solution. Thus, the client-server architecture may prevent a need to distribute the database. Thus, the client-server architecture may provide for ease of maintenance. The client-server architecture may allow to offer determination of the effect pigment data, identification of the effect pigment, analysis of a formulation, identification of the formulation and I or search for the formulation in the database as a service (aaS), for example knowledge as a service (KaaS).

[0034] In another aspect, the disclosure relates to a use of effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object as generated according to the 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0035] 7 methods for at least one of identifying the effect pigment, analyzing a formulation of the coating and producing the coating.

[0036] The use of the effect pigment data may provide for improved identification of the effect pigment in the coating with a high accuracy, improved analysis of the formulation of the coating and I or production of the coating. The use of the effect pigment data may also allow to new services such as knowledge as a service (KaaS).

[0037] EMBODIMENTS

[0038] There is a need to allow for simple identification of effect pigments in coatings.

[0039] An object of the present disclosure is to provide a method for improved determination of effect pigment data associated with an effect pigment comprised in a coating.

[0040] In the following, embodiments of the present disclosure will be outlined by ways of examples. It is to be understood that the present disclosure is not limited to said embodiments and I or examples.

[0041] In an embodiment, the computer-implemented method further comprises: receiving, for generating the at least one indication mark, user input indicating the at least one effect pigment pixel.

[0042] The method may provide for identifying the effect pigment based on a user-provided selection such as a manual selection. Thus, the method may provide for identifying the effect pigment in the coating with an even higher accuracy.

[0043] In another embodiment, the computer-implemented method further comprises: displaying the image data on a display; and detecting the user input in response to displaying the image date.

[0044] In another embodiment, the computer-implemented method further comprises: generating the at least one indication mark by applying at least one of a point, a region of interest, a minimum bounding box and a minimum bounding rectangle around the at least one effect pigment pixel.

[0045] The method may provide for identifying the effect pigment having a low concentration and / or a low occurrence in a coating. The method may provide for identifying the effect pigment even when having a rare presence and / or a rare use in coatings. Additionally or alternatively, the 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0046] 8 method may provide for identifying the effect pigment based on an automatic selection. Moreover, the method may provide for identifying the effect pigment using image segmentation methods and I or deep learning such as Segment Anything (Meta Al, April 2023). Thus, the method may provide for identifying the effect pigment in the coating with an even higher accuracy.

[0047] In another embodiment of the computer-implemented method, segmenting the coating image data comprises: using a data-driven segmentation model trained on segmentation data sets comprising image data of a plurality of elements with associated indication marks. The plurality of elements may be a plurality of objects, for example.

[0048] In another embodiment of the computer-implemented method, generating the at least one effect pigment class comprises at least one of: determining a candidate effect pigment and identifying the determined candidate effect pigment as the at least one effect pigment class, and determining a plurality of votes for a plurality of candidate effect pigments and identifying the candidate effect pigment of the plurality of candidate effect pigments having at least one of a maximum number of votes and a highest averaged probability of the plurality of votes as the at least one effect pigment class; and the candidate effect pigment has at least one of a maximum occurrence within a plurality of effect pigment classes comprising the at least one effect pigment class and a maximum probability within the plurality of effect pigment classes. The highest averaged probability may be a highest accumulated probability.

[0049] In another embodiment, the computer-implemented method further comprises: analyzing at least one of occurrences and probabilities within the plurality of effect pigment classes; and providing a result of the analysis to a user.

[0050] In another embodiment, the computer-implemented method further comprises determining a vote for a candidate effect pigment by applying a multi-class classification method and a softmax activation to the effect pigment data.

[0051] In another embodiment, the computer-implemented method further comprises identifying the effect pigment comprised in the coating.

[0052] In another embodiment, the computer-implemented method further comprises determining at least one of a color, a texture, at least one match criterion, at least one color match criterion, at least one texture color match criterion and at least one reflectance match criterion of the coating. 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0053] 9

[0054] In another embodiment, the computer-implemented method further comprises determining statistics of the at least one effect pigment class.

[0055] In another embodiment, the computer-implemented method further comprises determining a formulation of the coating.

[0056] In another embodiment, the computer-implemented method further comprises identifying the formulation in a database comprising a plurality of formulations.

[0057] In another embodiment, the computer-implemented method further comprises searching the database for the formulation, using at least one of the at least one match criterion, the at least one color match criterion, the at least one texture color match criterion and the at least one reflectance match criterion and the statistics.

[0058] In another embodiment, the computer-implemented method further comprises generating a statistics histogram, using the statistics.

[0059] In another embodiment of the computer-implemented method, the coating data further comprise coating identifier associated with the coating.

[0060] In another embodiment of the computer-implemented method, the coating data further comprise at least one of spectral data and reflectance data associated with the coating.

[0061] In another embodiment of the computer-implemented method, the coating data further comprise texture data associated with the coating.

[0062] In another embodiment of the computer-implemented method, the coating data further comprise color data associated with the coating, wherein the color data comprise at least one of color values, RGB values and CIE L*a*b* values.

[0063] In another embodiment of the computer-implemented method, the coating image data comprise a set of image data respectively acquired from the coating under different illumination conditions.

[0064] In another embodiment of the computer-implemented method, the coating image data are microscopic image data. 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0065] 10

[0066] In another embodiment of the computer-implemented method, the at least one effect pigment class comprises at least one of an effect pigment type, an effect pigment category and an effect pigment property.

[0067] In another embodiment of the computer-implemented method, the coating comprises a plurality of different pigments, and the effect pigment is one of the plurality of different pigments.

[0068] BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In the following, the present disclosure is further described with reference to the enclosed figures. The same reference numbers in the drawings and this disclosure are intended to refer to the same or like elements, components, and I or parts.

[0070] Fig. 1 illustrates an exemplary flow chart of a computer-implemented method 10 for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object according to an aspect of the present invention;

[0071] Fig. 2 illustrates an exemplary flow chart of a computer-implemented method 20 for training a data-driven classification model for generating at least one effect pigment class by classifying segmented image data according to an aspect of the present invention;

[0072] Fig. 3 illustrates an exemplary diagram of an apparatus 30 for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object according to an aspect of the present invention;

[0073] Fig. 4 illustrates an exemplary flow diagram 40 according to an aspect of the present invention;

[0074] Fig. 5 illustrates an exemplary system architecture showing a first configuration of the classification module;

[0075] Fig. 6 illustrates an exemplary system architecture showing a second configuration of the classification module;

[0076] Fig. 7 illustrates an exemplary system architecture showing a third configuration of the classification module;

[0077] Fig. 8 illustrates an exemplary system architecture showing a fourth configuration of the classification module;

[0078] Fig. 9 illustrates an exemplary system architecture showing a fifth configuration of the classification module;

[0079] Fig. 10 illustrates exemplary preprocessing of training data for a classification module according to an aspect of the present invention;

[0080] Fig. 11 illustrates exemplary training of a classification module according to an aspect of the present invention; 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0081] 11

[0082] Fig. 12 illustrates exemplary effect pigment classes; and

[0083] Fig. 13 illustrates a multi-sample evaluation method 130 according to an aspect of the present invention.

[0084] DETAILED DESCRIPTION

[0085] The following embodiments are mere examples for implementing the method, system or application device disclosed herein and shall not be considered limiting.

[0086] In the claims as well as in the description the word “comprising” or “including” or similar wording does not exclude other elements or steps and shall not be construed limiting to the elements or steps lined out. The indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation or further elements may be included.

[0087] The term “region of interest” (ROI) represents a portion of an image on which to operate on. A region of interest may be represented as a segmentation mask, binary mask image, bounding box (BBOX), minimum bounding box (MBB), bounding rectangle or minimum bounding rectangle (MBR), for example.

[0088] The term “indication mark” represents an indication or a reference to a region of interest or the like. The terms “indication mark”, “identification mark”, “label mark” and “selection mark” may be used synonymously.

[0089] The term “segmented image data” represents a portion of the coating image data being associated with the effect pigment. The terms “segmented image data”, “image patch” and “sub-image data” may be used synonymously.

[0090] The terms “Segment Anything” (Segment Anything Model, SAM, Meta Al, April 2023) and “Segment Anything 2” (SAM 2, Meta Al, July 2024) represent segmentation techniques providing zero-shot segmentation of unfamiliar objects and images without additional training. For example, trained on 11 million images and 1.1 billion masks, SAM has “learnt” a general notion of what objects are, and it may be applied “out of the box” for automatic segmentation such as effect pigment detection in microscopic images. 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0091] 12

[0092] The term ..transfer learning" represents a concept, wherein the architecture of a pre-configured neural network is used, but a classification layer of the neural network has been deleted and replaced with a new, own classification layer. The classification layer is usually a fully connected layer.

[0093] The term “ensemble learning” represents a machine-learning (ML) technique that aggregates at least two learning techniques, for example, an (artificial) network and a regression model, in order to produce better predictions. The terms “ensemble learning” and “committee-based learning” may be used synonymously. Ensemble learning uses base elements as individual models. The terms “base element”, “base estimator”, “base learner”, “base model” and “base predictors” may be used synonymously. The base elements may be classified in weak elements, e. g. weak learners, and strong elements, e. g. strong learners. Base elements that perform little better than random guessing, i. e. > 50 %, may be referred to as weak elements. Base elements that provide high predictive performance, e. g. > 80 %, may be referred to as strong elements. By combining a plurality of base elements, ensemble learning may yield a lower overall error rate while retaining complexities and advantages of the base elements.

[0094] The term “ensemble evaluation” represents use of ensemble learning for data evaluation. For example, multiple individual sub-images, or “image patches”, of pigments, known to belong to the same pigment type, may be passed to the classification model. As a result, the pigment class with a maximum number of votes or a highest probability is returned.

[0095] The term “vote” represents a result of the classifications associated with a pigment for a particular sub-image, i. e. for each sub-image, the pigment having the highest probability.

[0096] The term “voting” represents combining of predictions from multiple different models such as classifiers or regressors, and making a final prediction based on a majority vote or an averaging. The majority vote is usually used with the classifier. The average is usually used with the regressors. The voting uses the same but data different models. The term “hard voting” represents a type of voting, wherein each model casts a vote, and a majority of votes is decisive. The term “soft voting” represents a type of voting, wherein each model gives a probability estimate, and the highest averaged probability is decisive.

[0097] The term “EfficientNet BO” represents a convolutional neural network (CNN) that has been trained on more than a million images from a database of the ImageNet (http: / / www.imaqe-net.org). The 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0098] 13 pre-trained network can classify images into 1000 object categories, such as keyboard, mouse, pencil and many animals. As a result, the network has learned rich feature representations for a wide range of images. However, the pre-configured EfficientNet BO may also be used for transfer learning.

[0099] For successful color matching, a precise knowledge of the effect pigment composition is necessary. Determination of the effect pigment composition is mainly a manual process. Sample coatings are observed under a microscope, and expert knowledge of a colorist is required to identify effect pigments in coating. Difficult cases require a sequential comparison of effect pigments in sample with existing drawdowns of pre-selected pigments, and a final selection of pigments by side-to-side comparison based on visual similarity of flakes.

[0100] Fig. 1 illustrates an exemplary flow chart of a computer-implemented method 10 for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object according to an aspect of the present invention.

[0101] The computer-implemented method 10 comprises the following steps: receiving 12 coating data and indication mark, segmenting 14 coating image data, classifying 16 segmented image data, and determining 18 effect pigment data.

[0102] At step 12, the computer-implemented method 10 receives coating data associated with the coating, and at least one indication mark associated with the effect pigment. The coating data comprise coating image data of at least a part of the coating. The coating image data comprise a plurality of pixels. The image data may have been obtained using an image sensor. The image sensor may be comprises in a camera, spectrophotometer, microscope or smartphone, for example. The image data may conform to a defined specification. The at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels. The at least one indication mark may have been obtained by a manual selection of the effect pigment in the coating image data, for example. A user may select the effect pigment by clicking, for example using a pointing device such as a mouse, on the effect pigment displayed on a display, for example. Additionally or alternatively, the user may select the effect pigment by drawing a bounding box, such as a minimum bounding rectangle, around the displayed effect pigment. For effect pigments having a low occurrence in the coating, manual selection allows to determine effect pigment data correspondingly having low occurrence in the coating data associated with the coating with high accuracy. Ensemble evaluation and accumulation of votes and I or probabilities may further improve accuracy. Additionally or alternatively, the at least one indication mark may have been obtained 231417WO01 - Secondary Filing Text

[0103] BASF Coatings GmbH

[0104] 14 by an automatic pigment detection. At least one effect pigment pixel is associated with the effect pigment in the plurality of pixels.

[0105] At step 14, the computer-implemented method 10 segments, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment. The segmented image data may be represented by an image patch of a single effect pigment, cropped by a bounding box.

[0106] At step 15, the computer-implemented method 10 generates at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data. The at least one effect pigment class may comprise an effect pigment type, effect pigment categories such as silver dollar (SD) and I or effect pigment property such as an effect pigment size or effect pigment color, for example. The data-driven classification model has been trained on classification data sets comprising labelled image data of a plurality of different pigments, such as different effect pigments. The classification may use EfficientNet BO, a softmax activation and I or sigmoid activation, for example. The classification may output the at least one effect pigment class with at least one associated probability. For example, the classification may output the at least one effect pigment class with a plurality of associated probabilities, each of which being associated with a pigment of the plurality of different (effect) pigments.

[0107] At step 18, the computer-implemented method 10 determines, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment. The method may output an indication of or a recommendation for the effect pigment comprised in the coating. The determination may comprise evaluation and I or statistically analyzing the plurality of associated probabilities. The evaluation may comprise ensemble evaluation and I or voting. The (ensemble) evaluation may comprise obtaining a plurality of weak predictions, for example using a plurality of weak predictors, and congregating the plurality of weak predictions and extracting the indication or recommendation. The extraction of the indication or recommendation may use voting.

[0108] Fig. 2 illustrates an exemplary flow chart of a computer-implemented method 20 for training a data-driven classification model for generating at least one effect pigment class by classifying segmented image data according to an aspect of the present invention.

[0109] The computer-implemented method 20 comprises the following step: applying 22 classification data sets. 231417WO01 - Secondary Filing Text

[0110] BASF Coatings GmbH

[0111] 15

[0112] At step 22, computer-implemented method 20 applies classification data sets comprising labelled image data of a plurality of different effect pigments in at least one coating to the data-driven classification model. The labelled image data are respectively associated with at least one of effect pigment type, effect pigment category, effect pigment property such as an effect pigment size or effect pigment color of the different effect pigments. The method aims to correlate the image data with the corresponding associated label data.

[0113] The computer-implemented method 20 may further comprise the following step: correlating 24 image data with associated label data. The image data may comprise sub-images or image patches of pigments. The label data may comprise pigment classes associated with the pigments.

[0114] The data-driven classification model may be implemented as CNN, for example. The data-driven classification model may comprise at least one feature-detection layer to process image data. The at least one feature-detection layer may be implemented as an EfficientNet such as an Effi- cientNet BO and / or a ResNet such as a ResNet 50, for example. Additionally or alternatively, the data-driven classification model may comprise at least one layer to process image meta data and I or at least final classification layer.

[0115] The training of the data-driven classification model may use microscopic images of effect pigment mass-tones, and / or a mixture with black pigment and I or white pigment. These images usually comprise more than one effect pigment of the same known effect pigment type. An image segmentation method can be used automatically to crop sub-images of known effects pigments types out of these images such that a target, for example the effect pigment type, is known for each of the sub-images. Thus, use of manual image annotation may be avoided.

[0116] The training of the data-driven classification model may use transfer learning and I or pre-defined weights. Using transfer learning and I or pre-defined weights for training the data-driven classification model may reduce training time and I or costs.

[0117] Fig. 3 illustrates an exemplary diagram of an apparatus 30 for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object according to an aspect of the present invention.

[0118] The apparatus 30 comprises an input device 32 and at least one computing node 34. The apparatus 30 may be configured to implement the method 10 described with reference to Fig. 1. The apparatus 30 may further comprise an output device (not shown). 231417WO01 - Secondary Filing Text

[0119] BASF Coatings GmbH

[0120] 16

[0121] The input device 32 is configured to receive coating data associated with the coating, and at least one indication mark associated with the effect pigment. The coating data comprise coating image data of at least a part of the coating. The coating image data comprise a plurality of pixels. The at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels. At least one effect pigment pixel is associated with the effect pigment in the plurality of pixels.

[0122] The at least one computing node 34 is configured to segment, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment.

[0123] The at least one computing node 34 is further configured to generate at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on data sets comprising labelled image data of a plurality of different effect pigments.

[0124] The at least one computing node 34 is configured to determine, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

[0125] Fig. 4 illustrates an exemplary flow diagram 40 according to an aspect of the present invention.

[0126] The flow diagram 40 comprises a pigment detection module 44, pigment classification module 46 and pigment recommendation module 48. The pigment classification module 46 may comprise a voting module 47.

[0127] The pigment detection module 44 receives an image 42, such as a microscopic image, depicting at least a part of a coating comprising an effect pigment. The (microscopic) image 42 represents coating image data comprised in coating data associated with the coating. The (microscopic) image 42 comprise a plurality of pixels. At least one effect pigment pixel in the plurality of pixels is associated with the effect pigment. The pigment detection module 44 further receives at least one indication mark (not shown) associated with the effect pigment. The at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels.

[0128] The pigment detection module 44 is configured to segment, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment.

[0129] The pigment classification module 46 is configured to generate at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data. The 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0130] 17 data-driven classification model has been trained on data sets comprising labelled image data of a plurality of different effect pigments. The pigment classification module 46 is further configured to determine, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

[0131] The pigment classification module 46 may comprise a voting module 47. The voting module 47 may be configured to obtain a plurality of weak predictions, for example using a plurality of weak predictors, and congregating the plurality of weak predictions and extracting the indication or recommendation using voting. Even for a relatively low predictive performance, e. g. just larger than 50 %, of the individual weak predictors, a vote of the plurality of weak predictors may provide for determination of the effect pigment data with high accuracy. The vote may be a majority vote or an average vote, for example.

[0132] The pigment recommendation module 48 may provide an indication of or a recommendation for the effect pigment comprised in the coating. Additionally or alternatively, the pigment recommendation module 48 may provide an indication of or a recommendation for a formulation of the coating.

[0133] Fig. 5 illustrates an exemplary system architecture showing a first configuration of the classification module. The classification module comprises a convolutional neural network (CNN) for classifying image data, such as microscopic image data. The image data may be provided in any suitable image format, including, but not limited, grayscale image data and RGB (red, green, blue) image data, for example, and I or any suitable depth including, but not limited, 8 bits and 32 bits, for example. The classification module has been trained to map the input image data associated with an effect pigment to an output, i. e. target. In this configuration, the input to the classification module comprises single image data, i. e. image data associated with a single image. The image data may correspond to an image representing the effect pigment. In this configuration, the output of the classification module comprises one output, for example, an effect pigment type.

[0134] Fig. 6 illustrates an exemplary system architecture showing a second configuration of the classification module. In this configuration, the input to the classification module comprises single image data, i. e. image data associated with a single image, and image meta-data associated with the single image. The image meta-data may correspond to additional input data comprising, for example, at least one of spectral data, texture data and color values of a spectrophotometric measurement. The image meta-data may potentially improve model accuracy of the classification 231417WO01 - Secondary Filing Text

[0135] BASF Coatings GmbH

[0136] 18 module. In this configuration, the output of the classification module comprises one output, for example, an effect pigment type.

[0137] Fig. 7 illustrates an exemplary system architecture showing a third configuration of the classification module. In this configuration, the input to the classification module comprises image data associated with more than one image and image meta-data associated with more than one image. For example, the input to the classification module may comprise image data associated with a first image and a second image, and image meta-data associated with the first image and I or second image. The first image may be obtained in a “brightfield” condition, and I or the second image may be obtained in a “darkfield” condition, for example. For example, the microscopic images may be taken under “brightfield” illumination and I or “darkfield” illumination. In this configuration, the output of the classification module comprises one output (target), for example, an effect pigment type. However, as the output may be obtained from multiple inputs representing multiple samples, merging predictions obtained from the multiple samples may provide for higher accuracy.

[0138] Fig. 8 illustrates an exemplary system architecture showing a fourth configuration of the classification module. In this configuration, the output of the classification module comprises more than one output, i. e. multiple outputs or targets. For example, the output may comprise a first output, second output and third output. The first output may be related to a pigment category, the second output may be related to a pigment property, and the third output may be related to a pigment type, for example.

[0139] Fig. 9 illustrates an exemplary system architecture showing a fifth configuration of the classification module. In this configuration, the input to the classification module comprises image data associated with more than one image and image meta-data associated with more than one image, the output of the classification module comprises more than one output.

[0140] Fig. 10 illustrates exemplary preprocessing of training data for a classification module according to an aspect of the present invention.

[0141] The exemplary preprocessing comprises input, pre-processing and target values.

[0142] The input comprises an image such as a microscopic image, of reduced mass-tones of effect pigments. 231417WO01 - Secondary Filing Text

[0143] BASF Coatings GmbH

[0144] 19

[0145] The pre-processing comprises detection of the effect pigments in the image and cropping subimages of individual effect pigments from the image. The detection of the effect pigments may use a segmentation technique such as Segment Anything. The pre-processing may further comprise isolating the effect pigments in the associated sub-images, i. e. removing features that are not related to the effect pigments. The effect pigments may be isolated using pigment masks, that match the respective effect pigment.

[0146] The target values comprise automatic annotations of the image, with effect pigment type, effect pigment category and I or effect pigment property, for example.

[0147] The training data comprises pairs of sub-images and target values for effect pigments in the image.

[0148] Fig. 11 illustrates exemplary training of a classification module according to an aspect of the present invention.

[0149] The classification module comprises a CNN such as the EfficientNet BO. The CNN receives training data comprising pairs of sub-images and target values for effect pigments. The training data may have been obtained using the exemplary preprocessing of training data described with reference to Fig. 10.

[0150] During a feed-forward process, the training data is applied to the CNN and the CNN outputs a softmax classification for current values of its set of weights. A loss function determines an error such as a cross entropy for the current output and the target values such as the effect pigment types.

[0151] During an error-backpropagation process, the weights of the CNN are adjusted in order to minimize the loss function.

[0152] Fig. 12 illustrates exemplary effect pigment classes. The classes for effect pigments may be subdivided into a plurality of subclasses such as Pigment Type and Category, Pigment Color and Pigment Size, for example. The subclass Pigment Type and Category may be subdivided into a plurality of sub-subclasses. The sub-subclasses of the subclass Pigment Type and Category may comprise Aluminum and Interference, for example. The sub-subclass Aluminum may comprise at least one of Cornflake, Silver Dollar (SD) and TCR®, for example. 231417WO01 - Secondary Filing Text

[0153] BASF Coatings GmbH

[0154] 20

[0155] The sub-subclass Interference may comprise at least one of (natural) Mica, Xirallic®, Chromaflair®, Colorstream® and Glass Flake, for example. The subclass Pigment Color may indicate a color of a corresponding effect pigment and may comprise at least one of Silver, White, Red and Blue, for example. The subclass Pigment Size may indicate a size of a corresponding pigment relative to a plurality of different pigments and may comprise at least one of extra small (XS), small (S), medium (M), large (L) and extra-large (XL), for example.

[0156] Fig. 13 illustrates a multi-sample evaluation method 130 according to an aspect of the present invention. The multi-sample evaluation method 130 comprises a multiple input ensemble 1310, classification model 1320, multiple outputs 1330, voting 1340 and output 1350.

[0157] The multiple input ensemble 1310 comprises a plurality of inputs. The multiple inputs may comprise image data and image meta data associated with the image data. For example, the image data may represent one or more effect pigments.

[0158] The classification model 1320 represents a weak model. The classification model 1320 may comprise a plurality of weak predictors. These weak predictors generate, based on the multiple inputs 1310, the multiple outputs 1330. The weak predictors may predict classes associated with the one or more effect pigments.

[0159] The multiple outputs 1330 comprises a plurality of outputs such as a plurality of predictions or a plurality of votes regarding predictions and I or probabilities of the classes.

[0160] The voting 1340 represents a strong model. The voting 1340 represents a multi-sampling evaluation.

[0161] The output 1350 comprises one or more classified effect pigments, based on a majority of the votes or a highest averaged probability of the votes.

[0162] REFERENCE NUMERALS

[0163] 10 method

[0164] 12 receiving coating data and indication mark

[0165] 14 segmenting coating image data

[0166] 16 classifying segmented image data

[0167] 18 determining effect pigment data

[0168] 20 training method 231417WO01 - Secondary Filing Text BASF Coatings GmbH

[0169] 21

[0170] 22 applying classification data sets

[0171] 24 correlating image data with associated label data

[0172] 30 apparatus

[0173] 32 input device 34 computing node

[0174] 40 flow diagram

[0175] 42 (microscopic) image

[0176] 44 pigment detection module

[0177] 46 pigment classification module 47 voting module

[0178] 48 pigment recommendation module

[0179] 130 multi-sample evaluation method

[0180] 1310 multiple input ensemble

[0181] 1320 classification model 1330 multiple outputs

[0182] 1340 voting

[0183] 1350 output

Claims

231417WO01 - Secondary Filing Text BASF Coatings GmbH22Claims1. A computer-implemented method (10) for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, the method comprising: receiving (12) coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels; segmenting (14), using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment; generating (16) at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on classification data sets comprising labelled image data of a plurality of different effect pigments; and determining (18), based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

2. The computer-implemented method of claim 1 , further comprising: receiving, for generating the at least one indication mark, user input indicating the at least one effect pigment pixel.

3. The computer-implemented method of claim 2, further comprising: displaying the image data on a display; and detecting the user input in response to displaying the image date.

4. The computer-implemented method of one of claims 1-3, further comprising: generating the at least one indication mark by applying at least one of a point, a region of interest, a minimum bounding box and a minimum bounding rectangle around the at least one effect pigment pixel.

5. The computer-implemented method of one of claims 1-4, wherein: segmenting the coating image data comprises: using a data-driven segmentation model trained on segmentation data sets comprising image data of a plurality of elements with associated indication marks.231417WO01 - Secondary Filing Text BASF Coatings GmbH236. The computer-implemented method of claims 1-5, wherein: generating the at least one effect pigment class comprises at least one of: determining a candidate effect pigment and identifying the determined candidate effect pigment as the at least one effect pigment class, and determining a plurality of votes for a plurality of candidate effect pigments and identifying the candidate effect pigment of the plurality of candidate effect pigments having at least one of a maximum number of votes and a highest averaged probability of the plurality of votes as the at least one effect pigment class; and the candidate effect pigment has at least one of a maximum occurrence within a plurality of effect pigment classes comprising the at least one effect pigment class and a maximum probability within the plurality of effect pigment classes.

7. The computer-implemented method of claim 6, further comprising: analyzing at least one of occurrences and probabilities within the plurality of effect pigment classes; and providing a result of the analysis to a user.

8. The computer-implemented method of claims 1-7, further comprising: determining a vote for a candidate effect pigment by applying a multi-class classification method and a softmax activation to the effect pigment data; identifying the effect pigment comprised in the coating; determining at least one of a color, a texture, at least one match criterion, at least one color match criterion, at least one texture color match criterion and at least one reflectance match criterion of the coating; determining statistics of the at least one effect pigment class; determining a formulation of the coating; identifying the formulation in a database comprising a plurality of formulations; searching the database for the formulation, using at least one of the at least one match criterion, the at least one color match criterion, the at least one texture color match criterion and the at least one reflectance match criterion and the statistics; or generating a statistics histogram, using the statistics.

9. The computer-implemented method of claims 1-8, wherein: the coating data further comprise coating identifier associated with the coating;231417WO01 - Secondary Filing TextBASF Coatings GmbH24 the coating data further comprise at least one of spectral data and reflectance data associated with the coating; the coating data further comprise texture data associated with the coating; the coating data further comprise color data associated with the coating, wherein the color data comprise at least one of color values, RGB values and CIE L*a*b* values; the coating image data comprise a set of image data respectively acquired from the coating under different illumination conditions; the coating image data are microscopic image data; the at least one effect pigment class comprises at least one of an effect pigment type, an effect pigment category and an effect pigment property; or the coating comprises a plurality of different pigments, and the effect pigment is one of the plurality of different pigments.

10. A computer-implemented method (20) for training a data-driven classification model for generating at least one effect pigment class by classifying segmented image data, the method comprising: applying (22) classification data sets comprising labelled image data of a plurality of different effect pigments in at least one coating to the data-driven classification model; wherein the labelled image data are respectively associated with at least one of effect pigment type, effect pigment category, effect pigment property, effect pigment size and effect pigment color of the different effect pigments.

11. A computer program element with instructions, which when executed on at least one computing node is configured to carry out the steps of the method of any one of claims 1-10.

12. An apparatus (30) for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, the apparatus comprising: an input device (32) configured to receive coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels; at least one computing node (34) configured to: segment, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment;231417WO01 - Secondary Filing TextBASF Coatings GmbH25 generate at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on data sets comprising labelled image data of a plurality of different effect pigments; and determine, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.

13. An apparatus for providing coating data associated with a coating on a surface area of an object, the apparatus comprising: an output device configured to provide the coating data associated with the coating to an apparatus for determining effect pigment data associated with an effect pigment comprised in the coating, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels.

14. A client-server architecture for determining effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object, wherein: a client is configured to provide coating data associated with the coating, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels; and a server is configured to: receive the coating data associated with the coating, and at least one indication mark associated with the effect pigment, wherein the coating data comprise coating image data of at least a part of the coating, wherein the coating image data comprise a plurality of pixels, wherein the at least one indication mark identifies at least one effect pigment pixel in the plurality of pixels, wherein at least one effect pigment pixel is associated with the effect pigment in the plurality of pixels; segment, using the at least one indication mark, the coating image data into segmented image data associated with the effect pigment; generate at least one effect pigment class by classifying, using a data-driven classification model, from the segmented image data; wherein the data-driven classification model has been trained on data sets comprising labelled image data of a plurality of different effect pigments; and determine, based on the at least one effect pigment class, the effect pigment data associated with the effect pigment.231417WO01 - Secondary Filing TextBASF Coatings GmbH2615. Use of effect pigment data associated with an effect pigment comprised in a coating on a surface area of an object as generated according to the method of any one of claims 1-9 for at least one of identifying the effect pigment, analyzing a formulation of the coating and producing the coating.

Images