Method for detecting faults in mechanical components by classifying text strings describing issues with said mechanical components

Abstract

The present invention concerns a method for detecting faults in mechanical components by classifying text strings describing issues with said mechanical components by combining deterministic, rule-based keyword extraction algorithm with deep learning models. It uses a Generative AI model to generate synthetic data, especially for underrepresented classes. The system consists of two main components: a probability map that implements a deterministic, rule-based keyword extraction algorithm by focusing on negations and relevant keywords, and a Transformer-based model that handles long-range dependencies in sequential data. The outputs from both models are combined using score-based aggregation, and an ensemble classifier is trained to integrate these outputs, leading to improved accuracy in handling diverse data. The invention also includes a system with modules for data collection, preprocessing, augmentation, training, and evaluation to ensure optimal classification performance.

Description

[0001] Method for detecting faults in mechanical components by classifying text strings describing issues with said mechanical components

[0002] Applicant: Brembo N.V.

[0003] Inventors: Brignoli Silvia, Ravasio Mattia, Supriya

[0004] Ramarao Prasanna

[0005] The present invention concerns a method for synthetic data generation to enhance a Machine Learning text classifier, in particular for detecting faults in mechanical components by classifying text strings describing issues with said mechanical components.

[0006] State of the art

[0007] Integrating generative Al (GenAI) into a classification pipeline is primarily done to enhance the model's accuracy. In our specific case this is achieved by expanding the training dataset and improving the model's predictive performance. The application of this approach is beneficial for any machine learning (ML) classification task where the data is unevenly distributed across different classes (i.e. unbalanced dataset) . Sometimes collecting more data can be expensive or even unfeasible in the worst cases; in this way new observations can be gathered within few seconds thanks to GenAI .

[0008] Introducing synthetic data into the pipeline has already been done [F.1.1, F.1.3] and has been shown to bring relevant increase in performances. In this article

[0009] Jacobacci & Partners / ANP the LLM used for data generation is different from the one used for classification, an approach that is worth adopting if one does not want to bring any bias into the procedure. The choice for using a GPT-based LLM as source of data generation was used in [F.1.2] , which showed great achievements.

[0010] Other approaches that make use of the same LLM as ours for classification [F.1.4] , lack the use of an ensemble model or data augmentation to boost the performance .

[0011] The articles mentioning an ensemble of models [F.1.5, F.1.6, F.1.7] are used for different classification tasks, that involve understanding if a piece of text is real or synthetic, which is not the aim of the present invention. However, there appears to be no articles propending for the use of an ensemble empowered with synthetic data. Moreover, there is no mention of an approach that combines logical negation filtering protocol and LLMs .

[0012] Patent document US2023121812A1 focuses on improving AT model training through automated data augmentation techniques, particularly for image-based machine learning. It addresses the limitations of manual data enhancement, such as being time-consuming and subjective, by introducing a system that autonomously identifies data deficiencies and applies suitable augmentation methods. The system includes a data analyzer that measures the dataset's volume and variation, and a data generator that selects augmentation techniques to enhance the dataset. By

[0013] Jacobacci & Partners / ANP combining geometric transformations (e.g., flip / rotate, cropping) , color transformations (e.g., de-coloring, color conversion) , and advanced techniques (e.g., SMOTE sampling, mixup) , it generates a robust and diversified dataset. Using this enriched dataset, the Al model is trained more effectively, reducing overfitting and improving performance. This process can be implemented in cloud computing environments, leveraging scalable resources for efficient data augmentation and training. The invention significantly accelerates AT model development, especially in machine vision, by automatically selecting and applying augmentation techniques, leading to faster, more accurate training outcomes. The problem with this approach is that it refers to images. Contrary to what one might think, image and text generation are quite different. The mentioned techniques (SMOTE, MixUp, ...) do not need to make sure that the generated images '"make 'sense". Differently, text augmentation techniques must include in their pipeline a phase, posterior to generation, that accounts for lexicon and semantic accuracy. Furthermore, it is essential to check the textual data afterwards to confirm that the sentences are coherent and meaningful consistent with the class they belong to.

[0014] Object and subject-matter of the invention

[0015] The object of the present invention is method for the synthetic data generation to enhance a Machine Learning text classifier, which solves the problems and overcome the drawbacks of the prior art.

[0016] Jacobacci & Partners / ANP The subj ect-matters of the present invention is a method according to the attached claims .

[0017] Detailed description of invention embodiments

[0018] List of figures

[0019] The invention will now be described by way of illustration but not by way of limitation, with particular reference to the drawings of the attached figures , in which :

[0020] - figure 1 shows a flow chart of an invention embodiment ;

[0021] - figure 2 shows in ( a ) a confusion matrix for ensemble classi f ication without synthetic data and in (b ) , a confusion matrix for ensemble classi fication with synthetic data included in the pipeline .

[0022] It is speci fied here that elements of di f ferent embodiments can be combined together to provide further embodiments without limits while respecting the technical concept of the invention, as the skilled person understands without problems from what has been described .

[0023] The present description also refers to the known art for its implementation, with regard to the detailed characteristics not described, such as for example elements of minor importance usually used in the known art in solutions of the same type .

[0024] When introducing an element it always means that it can be "at least one" or "one or more" .

[0025] Jacobacci & Partners / ANP When a list of elements or characteristics is listed in this description it is meant that the invention according to the invention "includes" or alternatively "is composed of" such elements.

[0026] When features are listed within the same sentence or bulleted list, one or more of the individual features may be included in the invention without connection to the other features in the list.

[0027] Two or more of the parts (elements, devices, systems) described below can be freely associated and considered as kits of parts according to the invention.

[0028] Embodiments

[0029] The invention aims to enhance the performance of a ML text classifier by addressing its shortcomings in handling smaller, underrepresented classes. To achieve this, GenAI is employed to create synthetic data, which is then incorporated into the model's training process. This approach effectively increases the diversity and volume of the training data, allowing the model to "see" more different examples. By improving the model's exposure to underrepresented classes, the ultimate goal is to boost the overall accuracy and effectiveness of the classification task.

[0030] The invention model combines multiple approaches, more precisely it is a stack of classifiers, consisting of two main components (base learners) :

[0031] - The first module is not even a ML model but has been built with the intention to implement a deterministic, rule-based keyword extraction

[0032] Jacobacci & Partners / ANP algorithm with respect to a sentence. It identifies and removes negations, exploits relevant keywords (established a priori with Frequency-Inverse Document Frequency, TF-IDF, embeddings) and other details, then each sentence is assigned to a specific class by incrementally increasing a likelihood score whenever a feature relevant for that class is detected. This module represents the benchmark of human performance if an operator were to perform the classification.

[0033] - The second component is a pre-trained LLM, from the Transformers set. For example, a very large neural network (NN) consisting of an encoder that processes the input text, a linear layer and finally a SoftMax layer that outputs the classification scores .

[0034] Both models select the class label by simply referring to the highest score computed. In the first model, the score is computed based on the keywords that the claim contains. In the second model, the score is the output of e.g. the Softmax layer. The SoftMax layer assigns a probability score to each possible class. These scores represent the likelihood of the input text belonging to each class. The key feature of the SoftMax function is that it normalizes these scores so that they sum up to 1, effectively making them probabilities.

[0035] The outputs are then combined by means of a scorebased aggregation to become the inputs of an ensemble model. This approach puts together a more human-like

[0036] Jacobacci & Partners / ANP behavior and a deep learning model, boosting overall accuracy .

[0037] The training of the models is done after having generated enough samples for the under-represented classes. A quite small but reasonable amount of data is generated to ensure sample quality and diversity. The generated samples are already labeled which means that each generated text is already associated to a specific class. A domain expert may then verifies for any potential error. The whole training set is then classified by the two base learners (the two components of an ensemble model, i.e. probability map and transformer (the second model, based on pre-trained LLM) ) and their outputs are used for the training of the ensemble model. The latter is a tree-based classifier using as inputs the outputs of the models above. By combining all the steps, an increase in performance is achieved by the invention.

[0038] Making reference to Fig. 1, in step 110 the raw data are collected from sources or memory. In subsequent step 120 a GPT model (for example) processes claims (e.g. for spell check and translation) . The raw data are typically (but not exclusivity) text claims that need to be processed because they refer to warranty claims and are collected worldwide. Each text claim is written by technicians or mechanics that describe faulty mechanical pieces covered by warranty and are paired to each piece of production when it gets sent back to headquarters.

[0039] In step 125, the raw data are attributed a ground truth. It is here recalled that "attributing ground

[0040] Jacobacci & Partners / ANP truth" refers to the process of assigning the correct, verified labels or values to a dataset that serves as the reference standard for training and evaluating machine learning models. Ground truth represents the most accurate data available. In supervised learning, this ground truth is used as the target output the model learns to predict, and it is important for evaluating how well the model performs by comparing its predictions against this known, accurate data. Its name derives from the fact that it maps claims (text data) into labels. Its representation shows the likelihood of different classes or outcomes across a given input space, often used in tasks like image segmentation, object detection, or classification. Each element (e.g., pixel in an image) in the probability map contains a value ranging from 0 to 1, indicating the model's predicted probability that the element belongs to a specific class or category (corresponding to a specific label) . The output data are called "real data".

[0041] A small portion of the real data are used to create synthetic data via GET in step 130 to produce synthetic data .

[0042] In a preferred embodiment, the data augmentation step 130 is implemented as an automated, adaptive control loop for dataset balancing. The system initially performs a statistical analysis of the class distribution within the collected raw data to calculate the instance frequency for each ground truth label. Based on this analysis, the system identifies specific "underrepresented" classes where the instance count

[0043] Jacobacci & Partners / ANP falls below a pre-set threshold value , indicating a class imbalance . Upon detecting such an imbalance , the system dynamically constructs a speci fic natural language prompt . This prompt is programmatically generated to include the speci fic label of the identi fied underrepresented class along with context parameters related to the mechanical component faults . This dynamically constructed prompt is then transmitted to the Generative AT model to produce synthetic text strings corresponding exclusively to that speci fic class . This process iterates automatically until the instance count for the underrepresented class meets the required threshold, thereby systematically restoring dataset balance .

[0044] The so generated synthetic data are input into step 210 . Real data from step 120 is instead directly input into step 210 .

[0045] In step 210 , the dataset of labelled real data is subdivided into test data in 215 and Training data 216 . The dataset of synthetic data is added to Training set in 216 but not to the Test set in 215 .

[0046] In step 220 , optionally the training data are preprocessed ( e . g . lemmati zation and stemming ) , involving preparing and refining the ground truth data to ensure they are accurate , consistent , and suitable for training or evaluating an Al model .

[0047] In step 310 , a probability map is constructed as first base learner .

[0048] Optionally, before constructing the probability map, in step 320 further pre-processing is performed to handle

[0049] Jacobacci & Partners / ANP phrases or sentences where negation can change the meaning and / or ensure that relevant keywords are correctly identified. In a specific case, this is done so that the (first) model can behave with a human-like way of reasoning. It analyzes each text string and, when a keyword is detected, it attributes a positive score to the corresponding class. Instead, if the keyword is preceded by a negation, it is discarded.

[0050] In step 330, the map (statistical classifier model) is evaluated based on the pre-processed data from step 215 or 320. The output are classes of the input data with associated scores that represent a confidence level. Specifically, the output consists of (at least) the first and second most relevant labels as well as their scores (probabilities) . The use of two outputs instead of one is done because it is common that in the same claim different types of failures are described. An example of claim is as follows: "Clutch has been bled up several times but keeps going weak. Slave cylinder found to be very creaky and noisy, so replaced. Then found clutch master cylinder to be taking air in, customer also reported fluid leak from clutch reservoir".

[0051] Parallelly, from step 220, a pre-trained Al transformer model is (further) trained in step 410 as generally known, which brings efficiency and capability to handle long-range dependencies in sequential data.

[0052] The data fed for training into the transformer model is taken preferably from a sentence tokenization, in step 420, of data from 220 / 216. Tokenization is generally known and converts textual data into structured input

[0053] Jacobacci & Partners / ANP that models can process. It allows handling of out-of- vocabulary words or rare terms by breaking them into familiar sub-words. Tokenization is important for capturing the semantic and syntactic meaning of the input text .

[0054] After training in 430, transformer model deployment is performed as usual in step 440, and classes are obtained for the input data (test) as well as associated probability scores.

[0055] In a preferred embodiment, the statistical classifier module (Steps G-I) and the Al Transformer module (Steps J-K) are executed on separate parallel processing threads or distinct processing cores. The statistical module, being computationally lightweight, completes execution rapidly, while the Transformer module, which is computationally intensive, executes concurrently. This parallel architecture minimizes the total latency of the fault detection system.

[0056] At this point, the outputs from model evaluations 330 and 440 are "compared" in step 510. The 2 models are evaluated on the test set (from 215) . This means that for each claim present in test set we compute the predicted label for each of the 2 models. Then we compare the predictions of the 2 models with the ground truth and we check which model performed better (510) .

[0057] In particular, the outputs of the base learners are combined by means of a score-based aggregation to become the inputs of the ensemble model, the scores / probabilities being those obtained by the two

[0058] Jacobacci & Partners / ANP models above. This way of operating is necessary when dealing with multiple classifiers.

[0059] In order to have a single final label, their output is be used in 610 to have a unique response. Indeed, the combined outputs are then used in a "stack of classifiers" 620 also known as stacked generalization or stacking. It is an ensemble learning technique that combines multiple machine learning models to improve predictive performance. Instead of relying on a single model, stacking leverages the strengths of various classifiers to make more accurate predictions by training a meta-model to integrate their outputs.

[0060] The ensemble model is therefore trained in 630 on the combined outputs, is used to obtained a final classification in 640, and the evaluated in 710 on test set from 215 to check if the ensemble model is bringing novelty (increased performance) in the classification or not. The comparison happens with some evaluation metrics: both from classical statistics (Precision, Recall and Fl score) and with newer and more adapt scores (G-mean score and Index of Balanced Accuracy (IBA) ) .

[0061] The transformer model adopted in a use case is in the BERT (Bidirectional Encoder Representations from Transformers) family. A pre-trained public model is suitable for this kind of task, since building one from scratch would require enormous costs in terms of time and computing power. In a preferred embodiment, the method addresses a technical incompatibility between the output signals of the two base learners. The statistical classifier model (probability map) generates discrete,

[0062] Jacobacci & Partners / ANP integer-based scores derived from the frequency of keyword occurrences and their associated weights. In contrast, the Al transformer model outputs continuous probability distributions (Softmax scores) representing the likelihood of class membership. To enable the subsequent ensemble model to process these two disparate and incompatible data types without introducing bias or scaling errors, the integer-based scores (A) from the statistical classifier are subjected to a signal normalization step. Specifically, the scores are normalized into a probability space [0,1] using the formula : 1,2 where Ai and A2 are the scores associated to the first and second labels, respectively. This normalization ensures that the input vectors fed into the ensemble model are mathematically homogeneous, thereby improving the stability and convergence of the ensemble training process .

[0063] The first model (probability map) does not need training. But the second model (transformer) , even though is a pre-trained model, must be trained in order to address the task. It is trained for a small amount (e.g. 5-6) of epochs. Then also the ensemble model needs a training.

[0064] In the context of supervised learning, a tree-based

[0065] Jacobacci & Partners / ANP algorithm built to deal with imbalance classes is taken from existing models. Post-processing of the ensemble model output may include: evaluation of all the models with a set of metrics and visually comparing confusion matrices to assess which model is succeeding the best. Since the problem deals with imbalanced datasets, ad hoc metrics are built to highlight which model is correctly classifying rare labels.

[0066] In Fig. 2, two confusion matrices are given. Confusion matrix is a valuable tool used to evaluate the performance of a classification model across different classes. It provides a visual representation of the model's predictions compared to the actual outcomes, on vertical axes we have the true labels while on horizontal axis we have the predicted ones.

[0067] Figure 2a displays how well the model performs using only the original training data. It highlights the number of correct and incorrect predictions for each class, helping to identify areas where the model may struggle, such as underrepresented classes. Figure 2b presents the confusion matrix for ensemble classification with synthetic data included in the pipeline. This matrix demonstrates the impact of incorporating synthetic data generated by GenAI on the model's performance. By comparing this matrix with the one in Figure 2 (a) , one can observe improvements in the model's ability to correctly classify instances, especially in classes that were previously underrepresented.

[0068] Preferably according to the invention, the method is performed on cloud computing. The calls of GPT services

[0069] Jacobacci & Partners / ANP can happen by means of API keys and can be directly integrated into the pipeline.

[0070] The invention is therefore also directed to a server for enhancing the performance of a machine learning classifier, comprising code means with modules corresponding to the various steps above.

[0071] Applications and advantages of the invention

[0072] The invention method allows to improve the model's exposure to underrepresented classes, boosting the overall accuracy and effectiveness of the classification task .

[0073] In particular, the method is very effective on text strings describing issues with mechanical pieces.

[0074] Bibliography

[0075] [F.1.1] Zhao, H.; Chen, H.; Ruggles, T.A.; Feng, Y.; Singh, D.; Yoon, H.-J. Improving Text Classification with Large Language Model-Based Data Augmentation. Electronics 2024, 13, 2535.

[0076] [F.1.2] Tom B. Brown; Benjamin Mann; Nick Ryder et al. Language Models are Few-Shot Learners 2020 arXiv: 2005.14165

[0077] [F.1.3] Balkus SV, Yan D. Improving short text classification with augmented data using GPT-3. Natural Language Engineering. Published online 2023:1-30. doi :10.1017 / S1351324923000438

[0078] [F.1.4] S. Abarna, J. I. Sheeba, S. Pradeep Devaneyan, An ensemble model for idioms and literal text classification using knowledge-enabled BERT in deep

[0079] Jacobacci & Partners / ANP learning, Measurement: Sensors, Volume 24, 2022, 100434, ISSN 2665-9174, (link) [F.1.5] Harika Abburi, Michael Suesserman et al. Generative Al Text Classification using Ensemble LLM Approaches 2023, arXiv: 2309.07755

[0080] [F.1.6] A Simple yet Efficient Ensemble Approach for AI- generated Text

[0081] Detection (https : / / acl anthology . org / 2023. gem-1.32 ) (Abburi et al., GEM-WS 2023)

[0082] [F.1.7] Abburi, H., Suesserman, M., Pudota, N., Veeramani, B., Bowen, E., Bhattacharya, S. (2023) . An Ensemble-Based Approach for Generative Language Model Attribution. In: Zhang, F., Wang, H., Barhamgi, M., Chen, L., Zhou, R. (eds) Web Information Systems Engineering - WISE 2023. WISE 2023. Lecture Notes in Computer Science, vol 14306. Springer, Singapore.

[0083] In the foregoing, the preferred embodiments have been described and variations of the present invention have been suggested, but it is to be understood that those skilled in the art will be able to make modifications and changes without thereby departing from the relevant scope of protection, as defined by the claims attached.

[0084] Jacobacci & Partners / ANP

Claims

CLAIMS1. A computer-implemented method for detecting faults in mechanical components by classifying text strings describing issues with said mechanical components, comprising :A. Collecting (110) raw data from sources or memory, the raw data comprising text strings describing issues with mechanical pieces;B. Preprocessing (120) the raw data to correct and make the data uniform, outputting real data;C. Assigning ground truth labels (125) to the real data, outputting labelled real data;D. Augmenting (130) the labelled real data from step C with the use of generative Al to create labeled synthetic data;E. Dividing (210) the labelled real data into training data (216) and test data (215) and adding the labelled synthetic data to the training set only;F. Preprocessing (220) a dataset comprising training data and test data of step E, outputting preprocessed labeled data; characterized in that the method comprises a hybrid classification pipeline including:G. Providing (310) a statistical classifier model configured to provide a probability map representing the likelihood scores of the different ground truth labels across the dataset constituted by the training data and test data from step F, based on the presence of a predefined sets of keywords;Jacobacci & Partners / ANPH. Further preprocessing (320) the preprocessed labeled data of step F to discard keywords preceded by negations ;I. Using (330) the statistical classifier model of step G on the preprocessed labeled data of step H to generate initial prediction labels for the two most relevant classes of the test data, obtaining a probability map for the test data;J. Training (410) a pre-trained Al transformer model on the training dataset from step F, thus obtaining a trained Al transformer model;K. Classifying (430) the test data from step F by the trained Al transformer model, outputting labels data with associated probability scores;L. Combining (510) outputs from both the statistical classifier model (310) in step I and the trained Al transformer model (430) in step K by aggregating the highest of said likelihood scores in the probability map and said associated probability scores, thus obtaining combined outputs;M. Training (630) an ensemble model (620) on the combined outputs of step L, obtaining a trained ensemble model for final classification of the mechanical faults; wherein steps G-I and J-K are performed in parallel.

2. The method of claim 1, wherein the following two steps are performed after step M:N. Obtaining (640) classified data by using the trained ensemble model on the test data from step H;Jacobacci & Partners / ANP0. Evaluating (710) the performance of the ensemble model against the test data from step H;3. The method of claim 1 or 2, wherein in step L the score-based aggregation (510) combines the outputs of the probability map and the transformer model by selecting a ground truth label based on the highest score .

4. The method of any one claim 1 to 3, wherein in step L the aggregation (510) is made by means of a tree based machine learning model.

5. The method of any one claim 1 to 4, wherein in step J the training dataset is preliminarily tokenized (420) to output a tokenized training dataset, whereon the transformer model is trained.

6. The method of any one claim 1 to 5, wherein step F is executed by performing lemmatization and stemming.

7. The method of any one claim 1 to 6, wherein in step B is executed by performing spell checking and translation .

8. The method of any one claim 1 to 7, wherein the likelihood scores of step G are calculated as the highest score normalized to the sum of the two highest scores respectively associated to the first and second labels.Jacobacci & Partners / ANP9. The method of any one claim 1 to 8 , wherein step L is performed by a tree-based machine learning model .10 . The method of any one claim 1 to 9 , wherein in the combining step ( L ) , the likelihood scores from the statistical classi fier model are normali zed according to the formula p± = A / (Ai + A2 ) , where i=l , 2 and Ai and A2 are the scores associated with the first and second most relevant labels , before being input into the ensemble model .11 . The method of any one claim 1 to 10 , wherein the augmenting step ( D) is performed by submitting speci fic natural language prompts to a Generative AI model , said prompts configured to request the generation of synthetic text strings corresponding exclusively to ground truth labels identi fied as underrepresented in the collected raw data .12 . Computer program including code means configured to , when executed by a processor, cause the processor to perform the method steps of any one claim 1 to 11 .

13. A non-transitory computer-readable medium storing instructions that , when executed by a processor, cause the processor to perform the method steps of any one claim 1 to 11 .14 . A server for detecting faults in mechanical components by classi fying text strings describing issuesJacobacci & Partners / ANPwith said mechanical components , comprising code means with :- A collection module configured to collect ( 110 ) raw data from sources or memory, the raw data comprising text strings describing issues with mechanical pieces ;- A preproces sing module configured to pre-process( 120 ) the raw data to correct and make the data uni form, outputting real data ;- A ground truth labels assignation module ( 125 ) configured to assign ground truth labels ) to the real data, outputting labelled real data ;- An augmentation module configured to augment ( 130 ) the real data to create synthetic data by using a generative Al ;- A division module ( 210 ) configured to divide the preprocessed labelled real data into training data and test data and to add the labelled synthetic data to the training set only;- A pre-processing module configured to pre-process( 220 ) the dataset comprising training data and test data, outputting preprocessed labeled data ; characteri zed in that the server further comprises a hybrid classi fication pipeline including :- A probability map module configured to provide( 310 ) a statistical classi fier model configured to provide a probability map representing the likelihood scores of the di f ferent ground truth labels across the dataset constituted by the training data and test data from the preprocessedJacobacci & Partners / ANPlabeled data, based on the presence of a predefined sets of keywords;- A further processing module configured to preprocess (320) the dataset to discard keywords preceded by negations;- A classification module configured to use (330) the statistical classifier model on the preprocessed labeled data to generate initial predictions for the two most relevant classes of the test data, obtaining a probability map for the test data;- A transformer training module configured to train(410) a pre-trained Al transformer model on the training data, and thus to obtain a trained Al transformer model;- A transformer classification module configured to classify (430) the test data by the trained Al transformer model, outputting data with associated probability scores;- A combination module configured to combine (510) outputs from both probability map (330) and the transformer model (430) using aggregation (510) based on said probability map and said associated probability scores, thus obtaining combined outputs ;- An ensemble training module configured to train an ensemble model (630) on the combined outputs and to obtain a trained ensemble model for final classification of the mechanical faults; wherein the probability map module and the classification module, on the one hand, and theJacobacci & Partners / ANPtrans former training module and the trans former classi fication module , on the other hand, are operated in parallel before the operation of the combination model .Jacobacci & Partners / ANP

Images