Method and system for recognition of an object in an image

The client-server system with selective anonymization using VFM masks and markers addresses data privacy concerns in AI model training for PCBs, ensuring secure and accurate fault detection by protecting sensitive design data and preventing unauthorized duplication.

WO2026234012A1PCT designated stage Publication Date: 2026-11-12SIEMENS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/062961
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-08
Filing Date
2026-05-05
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing methods for training AI models on high-resolution images of proprietary PCB designs face challenges in protecting sensitive design data from reconstruction attacks while maintaining model accuracy, particularly in federated learning scenarios where uniform data protection measures can compromise precision.

Method used

A client-server system employs a pre-trained Vision Foundation Model (VFM) to generate local masks and markers for selected sensitive areas, enabling selective anonymization and training local recognition models through federated learning, ensuring only masked images are used for model updates, thus preserving data privacy and accuracy.

Benefits of technology

This approach enhances system security and reliability by protecting sensitive PCB design data, preventing unauthorized duplication, and maintaining high-quality training data integrity without compromising fault detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026062961_12112026_PF_FP_ABST
    Figure EP2026062961_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a computer-implemented method for the recognition of an object in an image, which is performed by a client-server system having a server and connected clients, wherein the following steps are carried out: a) providing a respective local set of images featuring the object to the respective clients, b) providing a pre-trained basic model for object recognition, wherein the model is designed to generate a mask and a marker for a selected region of an image, c) defining at least one respective selected region in the images from the respective local set of images, which is to be protected, d) generating a respective local mask for the at least one selected region and applying the respective local mask to the images, e) generating a respective local marker for the at least one selected region with the aid of the basic model and labeling the images with the respective local marker, f) training a respective local recognition model on the basis of federated learning with the respective local masked and labeled images, and aggregating the respective local recognition models into a global recognition model on the server, and distributing the global recognition model from the server to the clients, g) providing a further image featuring the object to a client selected from the clients, h) recognition of the object in the further image with the aid of the global recognition model received by the selected client.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method and system for recognizing an object in an image

[0002] The invention relates to a computer-implemented method and a system for recognizing an object in an image.

[0003] The invention further relates to a computer program product.

[0004] In the context of visual inspection of products such as the manufacture of printed circuit boards (PCBs), artificial intelligence (AI) models can be used, for example, to detect anomalies in or on a circuit board or in components arranged and attached to it, such as electronic components or resistors.

[0005] It is advantageous to use a model for this inspection task that is based on a large training dataset.

[0006] However, the availability of suitable training datasets may be limited, and methods for anonymized model training are preferred, such as the concept based on federated learning (FL).

[0007] FL allows multiple devices to train machine learning models together without sharing raw data.

[0008] However, it can still be difficult to reconcile detailed data analysis with data privacy concerns, because when using FL there may be a risk where model parameters can be reverse-engineered to reconstruct the training data, which in this case contains high-resolution images of proprietary PCB designs.

[0009] This risk is problematic because it could expose sensitive design details to unauthorized persons and endanger the company's intellectual property, but it can also provide insights into system security and reliability and could therefore be adversely affected by perpetrators.

[0010] If competitors or malicious actors can reconstruct these images, they could potentially replicate the company's unique designs and processes and market manipulated circuits that, for example, can be tapped or lack sufficient protection mechanisms.

[0011] Therefore, protecting training data from reconstruction attacks is crucial to maintaining the company's competitive advantage. In the visual inspection of printed circuit boards, the high-resolution images required for defect detection often contain sensitive design information.

[0012] It is a challenge to protect this proprietary data while simultaneously ensuring that the composite model remains accurate.

[0013] Anonymizing images to protect privacy can impair the model's ability to detect subtle flaws.

[0014] Therefore, manufacturers must carefully manage this compromise to protect intellectual property while maintaining a high level of effectiveness in the inspection process.

[0015] In the state of the art, a so-called "granular control" over data disclosure is known, in which current approaches such as "differential privacy", secure aggregation and homomorphic encryption primarily focus on anonymizing the entire data set or securing the data during transmission and computation.

[0016] However, they lack the ability to selectively anonymize certain parts of the data, which could be crucial in scenarios such as printed circuit board manufacturing, where some data segments are more sensitive than others.

[0017] Furthermore, the state of the art includes a so-called "balance between precision and data protection", in which uniform data protection measures are often applied to all data points, which can lead to a loss of accuracy, especially for parts of the data that are not so sensitive.

[0018] This can be particularly detrimental in applications such as visual inspection, where the model needs to analyze high-resolution images with high precision.

[0019] A vision language model (VLM) is a basic model of artificial intelligence, a so-called "foundation model", and is trained with a large dataset of images, where the images are paired with textual descriptions, thus enabling visual content to be linked with textual concepts.

[0020] VLMs can perform various tasks such as image classification, object detection, and tagging without requiring task-specific training.

[0021] One example of a popular open-source VLM is CLIP, short for "Contrastive Language-Image Pretraining". This approach has various use cases in the manufacturing industry, such as semantic search, where manufacturers can use VLMs to perform semantic searches in large image databases, thus facilitating the finding and retrieval of relevant images of parts or products based on descriptive queries.

[0022] Furthermore, there are applications for efficient training data generation, where VLMs can accelerate the creation of training datasets for other machine learning models used in manufacturing by generating labels and annotations through textual descriptions.

[0023] Furthermore, there are applications for automated documentation where VLMs can help in the creation and updating of documentation by automatically generating descriptions or tags for images of machines, components and processes, which facilitates the maintenance of accurate records.

[0024] The application of VLMs in manufacturing is associated with several challenges, such as data and domain gaps, because vision language models require large and diverse datasets to learn effectively.

[0025] If the training data does not cover certain areas or nuances that are relevant to a particular industry, it can be difficult to generalize the model or to perform well on these specialized tasks.

[0026] In the state of the art, VLMs are known, for example, for object recognition in a digital image, as in the publication Dieu-Thu Le, Jasper Uijlings, Raffaella Bernardi, “Exploiting language models for visual recognition”, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 769-779.

[0027] Furthermore, VLMs are used in the prior art, for example, to generate

[0028] Labels are known as markings (English: “label”), as for example in the publication Junsu Kim, Yunhoe Ku, Jihyeon Kim, JunukCha, Seungryul Baek, “VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model”, Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pages 4170-4181.

[0029] In the publication Li Qilong et al., "On mask-based image set desensitization with recognition support," APPLIED INTELLIGENCE, Vol. 54, No. 1, December 14, 2023, pages 886-898, XP093296788, a method for masking images is described. This method uses a federated approach to generate a mask model, taking data privacy aspects into account, and employs a "Feature Selection MaskNet" as a model fitting method to improve performance based on the masked images. However, a disadvantage of this approach is that it generates a central, federated mask model in addition to another model for object recognition, making it an overall suboptimal and inefficient solution.

[0030] In the publication Wu Meihan ET AL: "EFTViT: Efficient Federated Training of Vision Transformers with Masked Images on Resource-Constrained Edge Devices", arXiv.org, November 30, 2024, XP093296912, a hierarchical federated framework is described that uses masked images to enable efficient training with Juli parameters on resource-constrained edge devices, thus providing significant benefits for training with heterogeneous data.

[0031] However, masking the images in a simple and efficient way is still disadvantageous and time-consuming.

[0032] The object of the invention is therefore to provide a solution in which a model based on artificial intelligence is trained efficiently and accurately within a client-server system with high system security.

[0033] The aim is to find a solution in which specifically selected data from large datasets are processed anonymously in the model in a particularly efficient manner, and traceability for these individual data points can be prevented, thereby improving system security and system reliability against unwanted external interference and manipulation attempts.

[0034] The problem according to the invention is solved by a computer-implemented method for recognizing an object in an image, which is executed by a client-server system with a server and connected clients, wherein the following steps are performed:

[0035] a) Providing a local set of images of the object to the respective clients,

[0036] b) Providing a pre-trained basic model, VFM, for object recognition based on artificial intelligence, wherein the model is configured to generate a mask and a marker for a selected area of ​​an image, to the respective clients,

[0037] c) Defining at least one selected area as a marker in the images from the respective local set of images to be protected, d) Generating a local mask for the selected area and applying the local mask to the images from the set of images using the base model, VFM, by the respective client, e) Training a local recognition model based on federated learning using artificial intelligence with the locally masked images from the respective local set of images by the respective client, aggregating the respective local recognition models into a global recognition model on the server, and distributing the global recognition model from the server to the clients, f) Providing another image containing the object to a client selected from among the clients.

[0038] g) Recognition of the object in the wider image using the received global recognition model on the selected client.

[0039] This makes it possible, for example, to use Vision Foundation Models (VFM), i.e., models based on foundation models (FM), for image segmentation and labeling, while Federated Learning ensures that sensitive PCB design data, i.e., a printed circuit board design with conductor tracks and layers and electronic components, remains protected during AI model training.

[0040] The adaptive masking algorithm offers a flexible solution to maintain privacy and system security for the PCB elements without compromising the accuracy of fault detection.

[0041] The increased system security results from improved protection against, for example, manipulation or unwanted duplication for an illegal spare parts market.

[0042] Locally selective anonymization within an image allows manufacturers to protect only the most sensitive parts of the training data, such as specific areas of PCB images that reveal proprietary designs, while leaving other areas intact. This ensures that machine learning models receive high-quality, unaltered data for training, thus preserving accuracy where it is most needed.

[0043] This allows for an optimized compromise between data protection and usability, because selective anonymization of data enables manufacturers to better manage the compromise between data protection and accuracy.

[0044] This approach allows for privacy protection without unduly impacting model performance, which is crucial in quality control processes where detecting subtle errors is essential. In other words, federated learning is integrated with targeted anonymization of parts of an image, whereby the entire global FL model masks sensitive image areas, with the respective local detection models being trained on already masked images.

[0045] Initially, decentralized data processing is carried out, in which the training process for the AI ​​model takes place on several edge devices at different manufacturing sites, the so-called clients, with each site having its own data set of PCB images.

[0046] The VFM and masking algorithms are applied locally on each client to preprocess and segment the images.

[0047] Subsequently, a local model training takes place, in which a local fault detection model is trained with the pre-processed and segmented data from each location.

[0048] This ensures that the sensitive data used never leaves the local site, thus protecting the privacy of proprietary PCB designs. The sensitive data in the local images undergoes a masking process within a limited image area to suppress or remove specific details. Only the masked images are used in local model training, meaning the local model only generates images containing non-sensitive data.

[0049] Then, a model aggregation takes place, in which each location sends the locally trained model updates to the central server at regular intervals.

[0050] The central server aggregates these updates to create a global detection model, also known as a VFM, which benefits from the collective knowledge of all local models. Finally, an iterative training process takes place, in which the updated global model is sent back to each location for further local training, thereby iteratively improving the fault detection capabilities at all locations without exposing sensitive data.

[0051] The pre-trained foundation model (VFM) is an artificial intelligence-based object recognition model, a so-called "foundation model" which was trained using numerous general image datasets and can generate masks and markers for an image processing task through appropriate parameterization, for example, with corresponding input prompts. The masks can, for example, cover a predefined image section or manipulate image data such as pixels, and the markers for the masks can identify image areas within an image. A selected area can be understood as a marker.

[0052] The marker indicates the location of the mask in the image, making the masking process itself more efficient and accurate, as the masking then only needs to be applied to a portion of the image, and not to the entire image, thus preserving the detail depth of the remaining image.

[0053] Furthermore, several sub-areas can be processed independently of each other, for example even with priorities for their further processing.

[0054] Masking methods are explained in more detail below.

[0055] In summary, the particular advantage of the aforementioned model separation is the simple and efficient way of generating the mask from a provided VFM on the respective local client.

[0056] A local mask model can significantly reduce data traffic from the client to the server, as no images need to be transmitted, in addition to the aforementioned data privacy benefits.

[0057] Furthermore, the technical availability of the system can be significantly improved, as no large amounts of data need to be transferred, and therefore resilience against technical interruptions can be achieved.

[0058] Subsequent federation of an already masked image simplifies object recognition and increases efficiency, as less image information needs to be analyzed during inference.

[0059] The technical effect of this is that anonymized images are available for decentralized training in a simple and particularly efficient way, and anonymized models are already aggregated on the server, which are then distributed to clients for use in calculating an inference.

[0060] In a further development of the invention, it is provided that the creation of the masking is carried out using provided value ranges for an importance with regard to the anonymization or masking of the area to be marked, at least one selected area in the image, wherein the value for the importance of this area in the image is also provided to the respective client.

[0061] This makes it easy to create a list of areas that should be marked.

[0062] By applying value ranges for masking, it can be ensured, for example, that only a limited area is analyzed and the computational effort for this process is limited, or that marking only occurs when there is sufficient confidence in the recognition accuracy of the respective area.

[0063] In a further development of the invention, it is provided that the provided value range for importance is assigned to a masking procedure to be carried out.

[0064] The value range for importance is provided to each client.

[0065] The importance of a mask or region, comparable to assigning a priority to a mask within a set of multiple masks, indicates, for example, whether the respective mask should be processed with high priority (i.e., first) during inference calculations. It also weights the temporal sequence of multiple masks to accelerate convergence during the calculation and thus improve accuracy. This ensures that only importance values ​​within a predefined range are actually considered, making the processing of the inference image more efficient. Furthermore, it allows for prioritization across multiple regions within the inference image, thereby accelerating convergence in the process.

[0066] The respective importance of a mask or a preset, which are defined for an image area, are additionally provided with the inference image.

[0067] For example, a masking procedure for the selected image area to be protected can be carried out by creating a "blur" effect, a "disorientation" effect, a "smear" or "wavy" effect, or by applying a "pixelation" or "color distortion" procedure in at least one selected area to be masked or protected.

[0068] "Pixelation" refers, for example, to a reduction in image resolution in the selected area in order to obscure details, which can occur in a locally limited image area.

[0069] A “color distortion” can be understood, for example, as a modification of the color or brightness values ​​of individual pixels or image areas in the selected area, or also as a change in the contrast of contours in the selected area of ​​the respective image.

[0070] This allows an efficient method to be selected with a sensitivity level appropriate to a specific image section, i.e., a locally limited area of ​​the image, and thus the masking calculation can be performed efficiently according to the importance. In a further development of the invention, it is provided that the generation of a marker is carried out using the importance value.

[0071] By applying value ranges for marking and / or masking, it can be ensured, for example, that only a limited area needs to be analyzed and the computational effort for this process is limited, or that a warning is issued if a maximum proportional marking or masking area, measured against the size of the respective image, is exceeded, or that marking or masking only takes place if there is sufficient confidence for the recognition accuracy by the VFM of the respective area.

[0072] In a further development of the invention, it is provided that the generation of a respective local mask and / or a respective local marking for the at least one selected area is carried out using the basic model, VFM.

[0073] This allows the mask and / or marking to be created in a particularly simple way, since the VFM is already present in the client.

[0074] In a further development of the invention, it is provided that prior to training, a respective local marker is generated for the at least one selected area, and the images from the respective local set of images are labelled with the respective local marker using the basic model, VFN, and the training is carried out with the respective locally masked and labelled images from the respective local set of images.

[0075] In other words, the optional generation of a local marker for at least one selected area using the base model before local training is provided, and the labeling of the images from the respective local set of images with the respective local marker is provided.

[0076] This ensures that additional markings from the VFM are generated in a particularly simple way, that the VFM is already present in the client, and which further increases the model accuracy.

[0077] The object of the invention is achieved by a client-server system for recognizing an object in an image, comprising a server and connected clients, each of which has a computing device with a processor and a memory, and the client-server system is configured to execute the method according to the invention.

[0078] In a further development of the invention, it is provided that the client-server system further comprises an imaging sensor which is configured to capture a manufactured product as an object using a digital image and to provide it to the method as a further image.

[0079] The problem according to the invention is solved by a computer program product comprising instructions which, when the program is executed by a computer, cause it to execute the method / steps of the method according to the invention. The invention is described in more detail with reference to an exemplary embodiment. The figures show in

[0080] Fig. 1 shows an embodiment of the method according to the invention in the form of a flowchart,

[0081] Fig. 2 shows an embodiment of the invention as a block diagram,

[0082] Fig. 3 shows an embodiment of the invention in the form of a pseudocode.

[0083] Fig. 1 illustrates the method for recognizing an object in a digital image.

[0084] The following steps are performed, with at least one step being implemented in a computer:

[0085] a) Providing a respective local set of images of the object to the respective clients C1-C3,

[0086] b) Providing a pre-trained basic model, VFM, for object recognition based on artificial intelligence, wherein the model is configured to generate a mask and a marker for a selected area of ​​an image, to the respective clients C1-C3,

[0087] c) Defining at least one selected area as a marker in the images from the respective local set of images, which is to be protected,

[0088] d) Generating a local mask for at least one selected area using the basic model, VFM, and applying the local mask to the images from the set of images.

[0089] d 1 ) Generating a respective local marker for the at least one selected area using the basic model, VFM, and labeling the images from the respective local set of images with the respective local marker, e) Training a respective local recognition model based on federated learning using artificial intelligence with the respective locally masked and labeled images from the respective local set of images, and aggregating the respective local recognition models into a global recognition model on server S, and distributing the global recognition model from server S to clients C1-C3,

[0090] f) Providing another image of the object to a client selected from among clients C1-C3,

[0091] g) Recognition of the object in the wider image using the received global recognition model on the selected client.

[0092] Step d1) is optional and serves to improve object recognition through the use of markers.

[0093] The creation of the marker and / or a mask can be done using provided value ranges for the importance of the area to be marked, or at least a selected area in the present image.

[0094] The importance concerns the anonymization of the area to be marked, or at least a selected area, i.e., marking the sensitivity of that image area which must not be passed on to other units in the client-server system.

[0095] Creating a mask can be done using the importance level, which can also affect, for example, the size of the area to be anonymized.

[0096] In other words, the area of ​​the image to be masked, i.e., anonymized or protected, can also be larger than at least one selected area in order to achieve even better "coverage".

[0097] For example, the masking process can be achieved by creating a "blur" effect in an image area, a local "disorientation" in the image, or an effect for "blurring" or "shaking" in the image area, for which a VLM or VFM can also be used.

[0098] Furthermore, at least one selected area can be masked by applying a “pixelation” or color distortion method.

[0099] The selected image area consists of a number of pixels, with the pixels usually defining a contiguous area.

[0100] The large language model can be an artificial intelligence-based model, in particular a basic model (FM), for example, an image-text model. It is not absolutely necessary that the provided pre-trained basic models (VFM) are identical on clients C1-C3, which are further processed as local models, for example, as LM1, LM2, LM3.

[0101] In other words, different pre-trained basic models, VFM, can be provided to the respective clients C1-C3.

[0102] The global recognition model can be referred to as GM.

[0103] The provided value ranges for the importance of a selected image area can optionally be assigned to a marking procedure to be executed.

[0104] The aggregation of models can be carried out, for example, by summing selected edges in the model in the aggregated model using statistical means, for example by the mean of selected edges of the models to be aggregated.

[0105] Fig. 2 shows a client-server system for recognizing an object in an image.

[0106] The client-server system has a server S and clients C1-C3, each comprising a processor and memory.

[0107] Accordingly, the client-server system comprises a processor and a memory to execute the method according to the invention, wherein parts of the method are preferably executed on the high-performance server S.

[0108] The system comprises a server S and connected clients C1-C3, each of which has a computing device with a processor and memory.

[0109] The client-server system is configured to execute the method according to the invention.

[0110] The client-server system further includes, for each client C1-C3, a connected imaging sensor CAM1-CAM3, which is set up to capture a manufactured product as an object using a digital image.

[0111] The described method can therefore be executed by a system for recognizing an object in a digital image, which includes a computing device with a processor and a memory.

[0112] A product manufactured by a production device can, for example, be transported into the field of view of the respective camera CAM1-CAM3 using a conveyor belt. The respective camera CAM1-CAM3 can thus capture the manufactured product as an object using a digital image, and by applying the aforementioned method, the object can be recognized in the image and the resulting data processed further.

[0113] Fig. 3 shows an embodiment of the invention in the form of a pseudocode.

[0114] It is evident that masking an area in the image can be achieved, for example, by creating "blurring", "disorientation", "smearing" or "shaking", as well as by pixelation, i.e., a reduction of the image resolution in the area, or also by color distortion.

[0115] Reference symbol list

[0116] C1-C3 Client

[0117] CAM1-CAM3 Camera

[0118] S Server

Claims

Patent claims 1. Computer-implemented method for detecting an object in an image, executed by a client-server system with a server (S) and connected clients (C1-C3), wherein the following steps are performed: a) Providing a respective local set of images of the object to the respective clients (C1-03), b) Providing a pre-trained basic model, VFM, for object recognition based on artificial intelligence, wherein the model is configured to generate a mask and a marker for a selected area of ​​an image, to the respective clients (C1-03), c) Defining at least one selected area as a marker in the images from the respective local set of images to be protected, d) Generating a local mask for the selected area and applying the local mask to the images from the set of images using the basic model, VFM, by the respective client (C1-C3), e) Training a local recognition model based on federated learning using artificial intelligence with the respective locally masked images from the respective local set of images by the respective client (C1-C3), and aggregating the respective local recognition models into a global recognition model on the server (S), and distributing the global recognition model from the server (S) to the clients (C1-C3). f) Providing another image of the object to a client selected from among the clients (C1-03), g) Recognition of the object in the wider image using the received global recognition model on the selected client.

2. Method according to the preceding claim, wherein the creation of the masking is carried out using provided value ranges for an importance with regard to the anonymization of the at least one selected area in the image to be marked, wherein the value for the importance of this area in the image is also provided to the respective client.

3. Method according to the preceding claim, wherein the provided value ranges for importance are assigned to a masking method to be performed, for example, generating an effect of a “blur”, generating a “disorientation”, generating an effect of a “blur” or “shaky”, or by applying a pixelation method or a color distortion method in at least one selected area to be masked.

4. Method according to the preceding claim, wherein the creation of a mask is carried out using the importance.

5. Method according to one of the preceding claims, wherein prior to training in step e) a respective local marker is generated for the at least one selected area, and the images from the respective local set of images are labeled with the respective local marker using the basic model, VFN, and the training in step e) is performed with the respective locally masked and labeled images from the respective local set of images.

6. A client-server system for recognizing an object in an image, comprising a server (S) and connected clients (C1-C3), each of which has a computing device with a processor and memory, and the client-server system being configured to execute the method according to any one of the preceding claims.

7. A client-server system according to the preceding claim, further comprising an imaging sensor (CAM1-CAM3) configured to capture a manufactured product as an object using a digital image and to provide it to the method as a further image.

8. Computer program product, comprising instructions which, when the program is executed by the system according to one of the preceding system claims, cause the system to execute the procedure / steps of the procedure according to one of the preceding procedure claims.