Vision inspection method based on coarse-to-fine patch level classification and system thereof

Abstract

A vision inspection method based on coarse-to-fine patch level classification and a system thereof according to one embodiment of the present disclosure relate to a vision inspection method based on coarse-to-fine patch level classification and a system thereof which simultaneously improve the accuracy and processing speed of vision inspection by coarsely detecting a region suspected of being defective from given vision data and finely classifying the detected region on the basis of a pre-built data pool.

Description

Vision inspection method and system based on coarse-to-fine patch level classification

[0001] The present disclosure relates to a vision inspection method and system based on coarse-to-fine patch-level classification. More specifically, the invention relates to a vision inspection method and system based on coarse-to-fine patch-level classification that simultaneously improves the accuracy and processing speed of vision inspection by detecting (coarse) areas suspected of defects in given vision data and classifying (fine) the detected areas finely based on a pre-established data pool.

[0002] Vision inspection technology aims to automatically detect external defects in products through computer vision technology such as deep learning.

[0003] Good and defective products must be classified with very high accuracy based on data obtained from sensors such as cameras, and fast processing speed is also very important as the processing time of the inspection process directly affects product production volume.

[0004] However, since vision inspection is usually performed at high resolution to detect even minute defects, inspecting the entire area of ​​a product in detail with a deep learning model can achieve high accuracy, but there are clear limitations in terms of processing time.

[0005] To resolve this trade-off between accuracy and processing time, methods have been attempted to first detect suspected defect areas through rule-based algorithms or feature comparison with good products, and then make a final judgment through a deep learning classification model as post-processing.

[0006] However, these existing deep learning classification models using post-processing methods have fundamental limitations.

[0007] The model is trained to classify into good or defective types based on the features of the entire accumulated training data.

[0008] While this learning method enables the model to effectively distinguish overall defect patterns, there is a possibility of performance degradation when it comes to precisely classifying subtle feature pattern differences between good and defective products.

[0009] This is because, in the process of generalizing to enable overall good type classification based on the training dataset, the model tends to strengthen its discriminability based on broader features rather than fine features between good and defective products.

[0010] In actual vision inspection environments, situations frequently occur where normal pattern changes are mistaken for defects, such as when the characteristic pattern of a deformed good product is classified as a defect due to minute differences.

[0011] Therefore, in order to overcome the limitations of existing technology and fundamentally improve the judgment accuracy of vision inspection, a new type of deep learning model capable of effectively distinguishing these fine features is required.

[0012] One embodiment of the present disclosure is devised to solve the problems of the prior art as described above, and aims to provide a vision inspection method and system based on a coarse-to-fine patch level classification that simultaneously improves the accuracy and processing speed of vision inspection by detecting (coarse) areas suspected of defects in given vision data and classifying (fine) the detected areas finely based on a pre-established data pool.

[0013] However, the technical problems to be solved by the present disclosure and the embodiments thereof are not limited to the technical problems described above, and other technical problems may exist.

[0014] A vision inspection method based on a coarse-to-fine patch level classification according to one embodiment of the present disclosure comprises: a step in which at least one processor of the computer accesses a data structure of at least one memory of the computer; - wherein the data structure includes target vision data which is data specifying an object to be inspected, and the at least one processor loads the target vision data from the at least one memory; the at least one processor extracts patch-specific features of the loaded target vision data; the at least one processor detects at least one suspected defect patch within the target vision data based on the extracted patch-specific features; and the at least one processor obtains a prompt support set based on the first detected suspected defect patch. - Herein, the prompt support set includes at least one sample patch having a similarity greater than or equal to a preset standard with the first detected suspected defect patch and a correct answer label matching the sample patch, and the at least one processor inputs the acquired prompt support set and the first detected suspected defect patch into at least one artificial intelligence model to secondarily detect at least one suspected defect patch within the target vision data; and the at least one processor displays the result of the secondary detection through an interface.

[0015] In another aspect, the step of extracting patch-specific features of the target vision data includes the step of estimating a geometric transformation relationship between the pre-set good product vision data and the target vision data, and the step of aligning the target vision data according to the coordinate system of the good product vision data based on the estimated geometric transformation relationship.

[0016] In another aspect, the step of extracting patch-specific features of the target vision data further includes the step of dividing the aligned target vision data into patch units of a certain shape and the step of extracting features for each divided patch.

[0017] In another aspect, the step of first detecting the suspected defective patch includes the step of measuring the distance between the extracted patch-specific features and the patch-specific features of the good product vision data, and the step of first determining the suspected defective patch based on the measured distance between features.

[0018] In another aspect, the above-mentioned at least one processor further comprises the steps of collecting at least one vision data, dividing the collected vision data into patches of a certain shape, assigning a correct label to each of the at least one divided patches, and building a data pool by matching and storing the assigned correct label and the patch corresponding to the correct label.

[0019] In another aspect, the above correct answer label is defined as a type that specifies a good product type, a defective product type, or a specific shape of a defect.

[0020] In another aspect, the step of acquiring the prompt support set includes, among the at least one patch included in the constructed data pool, detecting at least one patch having a similarity greater than or equal to a preset standard with the first detected suspected defective patch, and detecting a correct answer label that matches the at least one patch having a similarity greater than or equal to the preset standard.

[0021] In another aspect, the step of obtaining the prompt support set further comprises: a step of measuring the distance between the features of the first detected suspected defective patch and the features of the at least one patch included in the data pool based on a predetermined similarity search algorithm; a step of selecting at least one patch among the at least one patch included in the data pool in order of the smallest distance between the measured features; and a step of obtaining the prompt support set by detecting the selected at least one patch and a correct answer label matching the selected at least one patch.

[0022] In another aspect, the at least one processor further comprises the step of training the at least one artificial intelligence model, wherein the at least one artificial intelligence model is a pre-trained artificial intelligence model that predicts a correct label for a query patch to be examined based on at least one sample patch to be referenced and a correct label matching the sample patch, and the step of training the at least one artificial intelligence model comprises the step of randomly extracting at least one patch and single patch data to which the correct label is assigned from the data pool, the step of generating a training episode dataset based on the randomly extracted at least one patch and single patch data, and the step of training the at least one artificial intelligence model to predict a correct label for the single patch data by referencing the at least one patch to which the correct label is assigned based on the generated training episode dataset.

[0023] In another aspect, the step of secondarily detecting the suspected defective patch includes inputting the acquired prompt support set into the at least one artificial intelligence model as a reference target, and inputting the firstly detected suspected defective patch into the at least one artificial intelligence model as an inspection target.

[0024] In another aspect, the step of secondarily detecting the suspected defective patch includes: a step of predicting a correct label for the firstly detected suspected defective patch based on at least one sample patch having a similarity greater than or equal to a preset standard with the firstly detected suspected defective patch included in the prompt support set, and a correct label matching the sample patch; and a step of outputting the result of the secondary detection for the firstly detected suspected defective patch based on the predicted correct label.

[0025] In another aspect, the step of implementing the result of the second detection through an interface includes the step of generating a perceptible notification signal indicating the quality status of the object to be inspected based on the result of the second detection, and the step of providing the generated perceptible notification signal through the interface.

[0026] In another aspect, the step of implementing the result of the second detection through an interface includes the step of transmitting the result of the second detection to at least one of an external device and a system to implement a service based on the result of the second detection.

[0027] In another aspect, the step of implementing the result of the second detection through an interface includes the step of transmitting the result of the second detection to at least one of an external device and system to trigger a predetermined automated measure corresponding to the result of the second detection.

[0028] Meanwhile, a vision inspection system based on coarse-to-fine patch level classification according to one embodiment of the present disclosure comprises at least one processor; and at least one memory storing at least one instruction that performs the following when executed by the at least one processor; wherein the at least one instruction comprises: a step in which the at least one processor accesses a data structure of the at least one memory, - wherein the data structure includes target vision data which is data specifying an object to be inspected; a step in which the at least one processor loads the target vision data from the at least one memory; a step in which the at least one processor extracts patch-specific features of the loaded target vision data; a step in which the at least one processor first detects at least one suspected defect patch within the target vision data based on the extracted patch-specific features; and a step in which the at least one processor obtains a prompt support set based on the first detected suspected defect patch, - wherein the prompt support set includes at least one sample patch having a similarity greater than or equal to a preset standard with the first detected suspected defect patch and a correct label matching the sample patch, and the at least one processor... the obtained prompt The method includes a step of inputting a support set and the first detected suspected defect patch into at least one artificial intelligence model to secondarily detect at least one suspected defect patch within the target vision data, and a step in which the at least one processor performs the step of manifesting the result of the second detection through an interface.

[0029] In another aspect, a vision inspection system based on a coarse-to-fine patch level classification according to one embodiment of the present disclosure comprises: a plurality of neurons configured in an array comprising at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; wherein each of the plurality of neurons further comprises a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network that is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

[0030] In another aspect, a vision inspection system based on a coarse-to-fine patch-level classification according to one embodiment of the present disclosure comprises: a plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons, wherein each of the plurality of neurons further comprises an application-specific integrated circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through any one of the plurality of synapse circuits.

[0031] A vision inspection method and system based on a coarse-to-fine patch level classification according to one embodiment of the present disclosure detects (coarse) areas suspected of defects in given vision data and classifies (fine) the detected areas finely based on a pre-established data pool, thereby simultaneously improving the accuracy and processing speed of vision inspection. This overcomes the trade-off between speed and accuracy, which was a technical limitation of existing automated inspection, and enables the implementation of a judgment paradigm with high efficiency, speed, and precision.

[0032] However, the effects obtainable in this disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0033] FIG. 1 illustrates an example of a block diagram of a computing system implementing a vision inspection service based on a Coarse-to-Fine Framework according to one embodiment of the present disclosure.

[0034] FIG. 2 illustrates an example of a block diagram of a computing device implementing a vision inspection service based on a core-to-fine framework according to one embodiment of the present disclosure.

[0035] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing a vision inspection service based on a core-to-fine framework according to one embodiment of the present disclosure.

[0036] FIG. 4 illustrates an example of a flowchart for explaining a vision inspection method based on coarse-to-fine patch level classification according to one embodiment of the present disclosure.

[0037] FIG. 5 illustrates an example of a drawing for explaining a data pool according to one embodiment of the present disclosure.

[0038] FIG. 6 illustrates an example of a flowchart for explaining a method for first detecting a suspected defective patch based on target vision data according to one embodiment of the present disclosure.

[0039] FIG. 7 illustrates an example of a drawing for explaining a method of aligning target vision data according to one embodiment of the present disclosure.

[0040] FIG. 8 illustrates an example of a drawing for explaining a method of dividing target vision data into patches according to one embodiment of the present disclosure.

[0041] FIG. 9 illustrates an example of a drawing for explaining a method for extracting patch-specific features according to one embodiment of the present disclosure.

[0042] FIG. 10 illustrates an example of a drawing for explaining a method of obtaining a Prompt Support Set according to one embodiment of the present disclosure.

[0043] FIG. 11 illustrates an example of a drawing for explaining a method for secondarily detecting a suspected defective patch based on a prompt support set according to one embodiment of the present disclosure.

[0044] As the present disclosure is capable of various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present disclosure is not necessarily limited to what is depicted.

[0045] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0046]

[0047] [Exemplary system providing vision inspection services based on the Coarse-to-Fine Framework]

[0048] Hereinafter, an exemplary system for implementing a vision inspection service based on a Coarse-to-Fine patch-level classification (hereinafter referred to as a Coarse-to-Fine Framework-based vision inspection service) that simultaneously improves the accuracy and processing speed of vision inspection by detecting (Coarse) areas suspected of defects in given vision data and classifying (Fine) the detected areas finely based on a pre-established data pool will be described in detail with reference to the attached drawings.

[0049] FIG. 1 illustrates an example of a block diagram of a computing system implementing a vision inspection service based on a Coarse-to-Fine framework according to one embodiment of the present disclosure.

[0050] Referring to FIG. 1, a computing system (1000) implementing a vision inspection service based on the Coarse-to-Fine framework of the present disclosure includes a user computing device (110), a server computing system (130), and a training computing system (150), and each device and system is connected to communicate via a network (170).

[0051] A vision inspection method and system based on a coarse-to-fine patch level classification according to one embodiment of the present disclosure may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or implemented and provided by the user computing device (110) and the server computing system (130) in conjunction with each other.

[0052] In this embodiment, the user computing device (110) and / or the server computing system (130) can train a machine learning model (120 and / or 140) through interaction with a training computing system (150) that is communicatedly connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0053] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0054] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0055] - User Computing Device (110: User Computing Device)

[0056] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device and / or a tablet PC.

[0057] Additionally, in the embodiment, the user computing device (110) may further include a predetermined server computing device that provides a vision inspection service environment based on a Coarse-to-Fine framework.

[0058] This user computing device (110) includes at least one processor (111) and memory (112).

[0059] Here, the processor (111) of the user computing device (110) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0060] In particular, according to the embodiment, this processor (111) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which is a hardware technology for implementing a certain digital circuit.

[0061] Here, a Field Programmable Gate Array (FPGA) implementation can refer to a flexible digital circuit that is programmable according to user needs.

[0062] As an example, a field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results or state information of operations to support synchronized operation of the FPGA, programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits configurable according to user needs, and an input interface that receives signals from external devices or sensors and transmits them to internal circuits as a channel for receiving data from outside the FPGA.

[0063] Through the combination of the above components, a field-programmable gate array implementation can provide flexible and various types of digital circuits.

[0064] Meanwhile, an Application-Specific Integrated Circuit (ASIC) can refer to a custom integrated circuit that is fixedly designed to perform a specific use or function.

[0065] As an example, the application-dedicated integrated circuit may include a register, which is a small memory device for temporarily storing and managing data and supports the rapid processing of ASIC operations by storing intermediate calculation results or state information; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and coordinates the operation of the entire system by performing various operations or generating control signals when necessary; and an input block, which is an interface for receiving data from the outside, which receives data to be processed by the ASIC and transmits it internally, and receives various input data through connections with sensors or external devices.

[0066] Through the combination of the components mentioned above, an application-specific integrated circuit can perform specific purpose tasks in an optimized manner.

[0067] Returning to the point, the memory (112) of the user computing device (110) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data (113) and instructions (114) necessary for the at least one processor (111) to perform functional operations, such as training an artificial intelligence model or performing vision inspection based on a Coarse-to-Fine framework through the artificial intelligence model.

[0068] In one embodiment, the user computing device (110) can perform various deep learnings for a vision inspection service based on a Coarse-to-Fine framework by linking with a Deep-learning Neural Network.

[0069] Here, the deep learning neural network according to the embodiment may include a Convolutional Neural Network (CNN), R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiments described below, and the embodiments of the present disclosure do not limit or restrict such deep learning neural networks themselves.

[0070] At this time, according to the embodiment, the deep learning neural network may be installed directly on the server computing system (130) or operate as a separate device from the server computing system (130) to perform deep learning for the vision inspection service based on the Coarse-to-Fine framework.

[0071] Additionally, in one embodiment, the user computing device (110) can store at least one machine learning model (120).

[0072] For example, the user computing device (110) may be composed of various machine learning models, such as multiple neural networks (e.g., deep neural networks) that perform a vision inspection method based on a Coarse-to-Fine framework based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0073] For example, machine learning models may include linear regression, decision trees, random forests, gradient boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0074] Additionally, according to an embodiment, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for a vision inspection method based on a Coarse-to-Fine framework through a large-scale language model (LLM).

[0075] In one embodiment, a user computing device (110) receives at least one machine learning model (120) from a server computing system (130) via a network (170), stores it in memory (112), and then executes the stored machine learning model (120) by a processor (111) to perform vision inspection based on a Coarse-to-Fine framework.

[0076] In another embodiment, the user computing device (110) can provide a vision inspection service based on a Coarse-to-Fine framework to the user by performing operations through a machine learning model (140) including at least one machine learning model (140) in conjunction with a server computing system (130) and communicating related data externally.

[0077] For example, a user computing device (110) can perform a vision inspection service based on a Coarse-to-Fine framework in which a server computing system (130) provides an output for the user's input using a machine learning model (140) via the web.

[0078] Additionally, the artificial intelligence model can be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0079] Additionally, the user computing device (110) may include at least one input component (121) that detects user input.

[0080] For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of the user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone that detects the user's voice input, a button, a mouse and / or a keyboard, etc.

[0081] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or video obtained by an image sensor device (e.g., CMOS or CCD).

[0082] In addition, the image sensor can process a still image or video acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to a processor.

[0083] Additionally, the input component (121) can receive input from an external controller (e.g., mouse, keyboard, etc.) based on an interface module, and in this case, may include an external output device (e.g., speaker).

[0084] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0085] In addition, the external output device may include a display system that outputs various information related to a vision inspection service based on a Coarse-to-Fine framework as a graphic image.

[0086] Such a display system may be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0087] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described later.

[0088] -Server Computing System (130: Server Computing System)

[0089] The server computing system (130) can perform a series of processes to provide a vision inspection service based on a Coarse-to-Fine framework.

[0090] In detail, in an embodiment, the server computing system (130) can provide the Coarse-to-Fine framework-based vision inspection service by exchanging data necessary to enable the Coarse-to-Fine framework-based vision inspection service process to run on an external device such as a user computing device (110).

[0091] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application can run on a user computing device (110).

[0092] To this end, the server computing system (130) may include an application program, data and / or instructions, etc. for the application to operate, and may transmit and receive various data based thereon with the external device.

[0093] Additionally, the server computing system (130) includes at least one processor (131) and memory (132).

[0094] Here, the processor (131) of the server computing system (130) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0095] In particular, depending on the embodiment, such a processor (131) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which are hardware technologies for implementing a specific digital circuit. A detailed description thereof is omitted by applying the description of the FPGA and ASIC mentioned above.

[0096] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations, such as training an artificial intelligence model or performing vision inspection based on a Coarse-to-Fine framework through the artificial intelligence model.

[0097] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected to a network (170).

[0098] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer non-linear model as the machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0099] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, data store) which is a storage for continuously storing and managing raw data that forms the basis of a vision inspection service based on a Coarse-to-Fine framework.

[0100] Such data stores may include various forms of data storage, ranging from file systems to cloud storage. For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0101] - Training Computing System (150: Training Computing System)

[0102] The training computing system (150) includes at least one processor (151) and memory (152).

[0103] Here, the processor (151) of the training computing system (150) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0104] In particular, depending on the embodiment, this processor (151) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which are hardware technologies for implementing a specific digital circuit. A detailed description thereof is omitted by applying the description of the FPGA and ASIC mentioned above.

[0105] And the memory (152) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0106] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error (according to the framework illustrated in FIG. 3).

[0107] For example, such a model trainer (160) can perform backpropagation updates to one or more parameters of a machine learning model (120 and / or 140) for a vision inspection service based on a defined loss function.

[0108] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight devaluation, dropout and / or knowledge distillation, etc.) to improve the generalization ability of the machine learning model (120 and / or 140) being trained.

[0109] Additionally, the model trainer (160) can train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data of different forms, such as images, audio samples and / or text, for example. Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images and / or various other forms of images.

[0110] These training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0111] And the model trainer (160) includes computer logic that is utilized to provide the desired function.

[0112] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, the model trainer (160) may include a program file stored in a storage device, be loaded into memory (152), and be executed by one or more processors (151). In another embodiment, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0113] Network (170) includes, but is not limited to, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network and / or DMB (Digital Multimedia Broadcasting) network.

[0114] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP and / or FTP, etc.), encodings or formats (e.g., HTML and / or XML, etc.), and / or protection schemes (e.g., VPN, Secure HTTP and / or SSL, etc.).

[0115] FIG. 2 illustrates an example of a block diagram of a computing device implementing a vision inspection service based on a Coarse-to-Fine framework according to one embodiment of the present disclosure.

[0116] Including FIG. 2, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing application (e.g., Detection, Classification and / or Segmentation, etc.), a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application and / or a chat-bot application, etc.

[0117] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and by storing and operating the trained artificial intelligence model, it may provide output data according to a predetermined input data (in an embodiment, vision data, etc.).

[0118] Each application of the computing device (100) can communicate with a number of other components of the computing device (100), such as, for example, at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0119] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing a vision inspection service based on a Coarse-to-Fine framework according to one embodiment of the present disclosure.

[0120] Referring to FIG. 3, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application and / or a browser application. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., a common API across all applications).

[0121] The central intelligence layer may include a number of machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided for each application and managed by the central intelligence layer. In another embodiment, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model for all applications. In some embodiments, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0122] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 3, the central device data layer can communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0123] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0124]

[0125] [Method for Providing Vision Inspection Services Based on the Coarse-to-Fine Framework]

[0126] Hereinafter, a method for implementing a vision inspection service based on a Coarse-to-Fine framework, in which a computing system (1000) according to one embodiment of the present disclosure detects (Coarse) areas suspected of being defective in given vision data and finely classifies (Fine) the detected areas based on a pre-established data pool to simultaneously improve the accuracy and processing speed of vision inspection, will be described in detail with reference to the attached drawings.

[0127] FIG. 4 illustrates an example of a flowchart for explaining a vision inspection method based on coarse-to-fine patch level classification according to one embodiment of the present disclosure.

[0128] Referring to FIG. 4, a method for providing a vision inspection service based on a Coarse-to-Fine framework according to one embodiment of the present disclosure may include a model learning and building step including a step of learning and building a model for a vision inspection service based on a Coarse-to-Fine framework (S101); a step of acquiring target vision data (S103); a step of first detecting a suspected defect patch based on the acquired target vision data (S105); a step of acquiring a prompt support set based on the first detected suspected defect patch (S107); a step of secondarily detecting a suspected defect patch based on the acquired prompt support set (S109); and a step of providing a detection result (S111).

[0129] First, a computing system (1000) according to one embodiment of the present disclosure can learn and build a model for a coarse-to-fine vision inspection service according to a process according to step S101.

[0130] Specifically, a computing system (1000) according to one embodiment of the present disclosure can learn and build a model for a vision inspection service based on a Coarse-to-Fine Framework. (S101)

[0131] Here, the Coarse-to-Fine Framework according to the embodiment may refer to a stepwise inspection methodology that simultaneously improves the speed and accuracy of vision inspection by sequentially performing a step of rapidly detecting areas suspected of defects in given vision data (Coarse step) and a step of precisely determining the detected suspected areas by referring to similar samples in a pre-established data pool (Fine step).

[0132] FIG. 5 illustrates an example of a drawing for explaining a data pool according to one embodiment of the present disclosure.

[0133] Referring specifically to FIG. 5, in the embodiment, the computing system (1000) can 1) build a data pool.

[0134] Here, the data pool according to the embodiment may refer to a knowledge database referenced to precisely compare and determine suspected defective patches.

[0135] In the embodiments, such a data pool may be configured as a logical storage space, which may be physically implemented in various storage spaces such as file storage, databases, and / or non-volatile memory.

[0136] More specifically, the computing system (1000) can collect at least one vision data for a predetermined product (in an example, a product to be inspected).

[0137] At this time, according to the embodiment, the computing system (1000) can secure diversity of collected vision data by collecting various vision data including vision data of products that are ultimately judged to be good products, as well as clear good or defective products, and products that have slight external deformation.

[0138] Additionally, the computing system (1000) can divide the collected vision data into patches and then assign a correct label to each patch.

[0139] In this case, as an example, the computing system (1000) can assign the above-described correct answer label based on input from a predetermined user (e.g., an expert, etc.).

[0140] For example, the computing system (1000) can receive label information entered by a specific expert user who directly distinguishes and determines the type of each patch and assign the correct answer label.

[0141] In another embodiment, the computing system (1000) may be linked with a previously verified deep learning model (e.g., a classification model based on a Convolutional Neural Network (CNN)), obtain classification result information for each patch type whose reliability among the prediction results of the deep learning model meets a predetermined standard (e.g., above a specific threshold), and automatically assign this as the correct label for the patch.

[0142] Here, the above correct answer label can be defined as a detailed type that specifies various shapes, such as multiple types of good products, multiple types of defective products, or specific states (shapes) of defects, going beyond a type that simply distinguishes between good products (OK) and defective products (NG).

[0143] That is, the computing system (1000) can assign a correct answer label to each patch that specifies various shapes, such as multiple types of good products, multiple types of defective products and / or specific states (shapes) of defects, beyond simply distinguishing between good products (OK) or defective products (NG).

[0144] For example, the computing system (1000) may assign a correct answer label to each patch, defined as a specific type related to the state of various defects, such as 'Crack', 'Dent', and / or 'Other'.

[0145] Additionally, as shown in FIG. 5, the computing system (1000) can match the correct answer label assigned as above with the corresponding patch, vision data and / or features extracted from the vision data (e.g., feature vector, etc.) and store them in a physical storage space such as a database.

[0146] Thus, the computing system (1000) can build a data pool for a vision inspection service based on a Coarse-to-Fine Framework.

[0147] In addition, in the embodiment, the computing system (1000) can continuously update the data pool constructed as above by reflecting cases where new types of vision data are obtained during a future inspection process or judgment information from a specific user (e.g., an expert) is added thereto.

[0148] In addition, in the embodiment, the computing system (1000) can train a fine classification model.

[0149] Here, the fine-grained classification model according to the embodiment, unlike a general classification model, may refer to a model that is pre-trained through a predetermined method (e.g., meta-learning and / or in-context learning, etc.) to receive a 'query patch' (i.e., a patch suspected of being defective in the embodiment) which is the subject of inspection (discrimination) and a reference set of N (N>=1) similar sample patch-label sets (in the embodiment, a 'prompt support set') which are the subject of reference, and to compare and analyze the fine feature differences between them to make a final judgment.

[0150] Specifically, the computing system (1000) can load a predetermined deep learning model (e.g., a model based on a Transformer architecture including an attention mechanism) as a micro-classification model.

[0151] Additionally, the computing system (1000) can generate a single training episode dataset by randomly extracting N labeled patches and single patch data (i.e., a single patch excluding labels) from a data pool.

[0152] In other words, the computing system (1000) can randomly extract N patch-correct-label data and one patch data to form a single training episode data set.

[0153] Additionally, the computing system (1000) can use the learning episode data set generated as above to train the loaded microclassification model to predict the correct label of the remaining one patch data (e.g., the type to which the product belongs, etc.) by referring to the given N patch-correct label data according to a predetermined method (e.g., meta-learning and / or in-context learning, etc.).

[0154] Accordingly, the computing system (1000) can build a fine classification model that learns the ability to distinguish fine differences between similar samples, rather than simply memorizing specific types of products (e.g., good products or defective products) that are classified.

[0155] At this time, in the embodiment, the computing system (1000) can repeat the process of generating the above-described learning episode data set and training a micro-classification model through it, thereby enabling the corresponding micro-classification model to learn the ability to distinguish between fine feature differences between various types of patches.

[0156] In addition, a computing system (1000) according to one embodiment of the present disclosure can operate and provide a model learned and constructed according to step S101 described above, according to a process according to steps S103 to S111.

[0157] In detail, a computing system (1000) according to one embodiment of the present disclosure can acquire target vision data. (S103)

[0158] Here, the target vision data according to the embodiment may refer to data that converts an object to be inspected, to be determined as good or defective, into a digital form (e.g., a digital image, etc.) and specifies it.

[0159] In detail, as an example, the computing system (1000) can acquire target vision data as described above by linking with a predetermined camera and / or sensor, etc.

[0160] For example, a computing system (1000) can perform data communication with a camera (e.g., a high-resolution industrial camera installed on a manufacturing process line, a line scan camera, etc.) and / or a sensor (e.g., an X-ray sensor, an infrared sensor, etc.) mounted on a specific vision inspection device, and receive and acquire target vision data including still images, video frames and / or sensor measurements output from said camera and / or sensor, etc.

[0161] At this time, the computing system (1000) can store the target vision data obtained as above in at least one memory of the computing system (1000).

[0162] Accordingly, the above-mentioned at least one in-memory data structure may include target vision data that specifies the object to be inspected.

[0163] And at least one processor of the computing system (1000) can access the data structure of the at least one memory to load the target vision data and perform preparation for a subsequent step.

[0164] In addition, a computing system (1000) according to one embodiment of the present disclosure can detect a suspected defective patch based on acquired target vision data. (S105)

[0165] Specifically, in the embodiment, the computing system (1000) can perform a primary detection (coarse detection) that quickly determines a suspected defective patch within the target vision data.

[0166] Here, a patch suspected of being defective according to the embodiment may refer to a patch among a plurality of patches constituting target vision data that is presumed to have a possibility of being defective when the difference exceeds a predetermined standard (e.g., threshold value, etc.) when comparing features with a patch at the same location within pre-set representative good product data.

[0167] At this time, the representative good product data according to the embodiment refers to vision data of a product determined to be defect-free (i.e., good product vision data), which may mean data that is pre-set and stored to be used as a baseline for feature comparison when searching for a suspected defect patch.

[0168] In an embodiment, the computing system (1000) selects at least one of at least one good product vision data and stores it in advance in a predetermined storage space such as non-volatile memory, and can utilize it for alignment and / or feature comparison of target vision data described later, etc.

[0169] In detail, in an embodiment, the computing system (1000) can perform a patch-unit feature comparison between each of the pre-set K (K>=1) representative good product data and the target vision data to extract at least one suspected defective patch.

[0170] In the following description, for effective explanation, the explanation is based on an embodiment in which at least one suspected defect patch is extracted based on one representative good product data and target vision data; however, this is merely one embodiment and is not limited thereto. Depending on the embodiment, various embodiments may be possible, such as comparing a plurality of representative good product data and target vision data and extracting at least one suspected defect patch based thereon.

[0171] FIG. 6 illustrates an example of a flowchart for explaining a method for first detecting a suspected defective patch based on target vision data according to one embodiment of the present disclosure.

[0172] More specifically, referring to FIG. 6, in an embodiment, the computing system (1000) can align target vision data. (S201)

[0173] Generally, positional errors that occur when photographing a product for inspection (e.g., minute parallel translation, rotation, and / or size changes of the product) can cause unnecessary errors when measuring differences from good product characteristics, thereby degrading overall inspection performance.

[0174] To resolve these factors, the computing system (1000) according to the embodiment can align target vision data based on representative good product data.

[0175] More specifically, the computing system (1000) can estimate the geometric transformation relationship between the target vision data and the representative good product data, and based on this, transform the target vision data to fit the coordinate system of the representative good product data to perform alignment.

[0176] FIG. 7 illustrates an example of a drawing for explaining a method of aligning target vision data according to one embodiment of the present disclosure.

[0177] In an example, the computing system (1000) can extract at least one feature point for each of the target vision data and the representative good product data using a predetermined feature point extraction algorithm (e.g., SIFT, SURF, FAST9, SuperPoint, Munkres, LightGlue algorithm, etc.).

[0178] Additionally, the computing system (1000) can compare at least one extracted feature point and mutually match feature points at the closest positions to form a single data pair.

[0179] And the computing system (1000) can estimate a homography matrix H representing the position and rotation relationship between target vision data and representative good product data based on at least one configured data pair according to the following [Equation 1].

[0180] [Mathematical Formula 1]

[0181]

[0182] Here, x' in [Equation 1] may mean the x-coordinate on the representative good product data and y' may mean the y-coordinate on the representative good product data, and x may mean the x-coordinate on the target vision data and y may mean the y-coordinate on the target vision data.

[0183] Additionally, the computing system (1000) can obtain target vision data aligned to the same position as the representative good product data by applying the homography matrix estimated as above to the target vision data.

[0184] At this time, the computing system (1000) can effectively remove noise components (outliers) included in the matched data pairs and select the optimal matching pairs by using a predetermined optimization algorithm.

[0185] As a specific example, a computing system (1000) can calculate a temporary homography matrix by repeatedly randomly sampling some of the matching pairs using an optimization algorithm such as RANSAC (RANdom Sample Consensus), and finally estimate the optimal homography matrix that has the largest number of data (Inliers) supporting the temporary homography matrix among all matching pairs.

[0186] And the computing system (1000) can apply the homography matrix finally estimated as above to the target vision data to align it to the same position as the representative good product data.

[0187] In this way, in the embodiment, the computing system (1000) performs an alignment process that precisely corrects the position between the target vision data and the representative good product data, thereby minimizing false detections caused by position errors in the subsequent feature comparison step and improving the accuracy of detecting suspected defective patches.

[0188] Additionally, in the embodiment, the computing system (1000) can divide the target vision data into patches. (S203)

[0189] FIG. 8 illustrates an example of a drawing for explaining a method of dividing target vision data into patches according to one embodiment of the present disclosure.

[0190] In detail, referring to FIG. 8, the computing system (1000) can divide the target vision data into patches of a certain shape (e.g., a grid shape of 16x16 pixels or 32x32 pixels).

[0191] In an example, the computing system (1000) can divide target vision data into a plurality of patches such that each divided patch has a mutually exclusive region (i.e., to exclude overlapping regions), as in FIG. 8 (a).

[0192] In another embodiment, the computing system (1000) may divide the target vision data into a plurality of patches such that there is an area of ​​mutual overlap of a predetermined size (m) between adjacent patches, as shown in (b) of FIG. 8, in order to minimize feature distortion or detection errors that may occur at the edges between each divided patch.

[0193] Thus, the computing system (1000) can divide the target vision data into a total of k (k>=1) patches, as shown in (c) of FIG. 8.

[0194] Through this, the computing system (1000) can resolve the problem of excessive computational load and resource usage that occurs when processing the entire area of ​​vision data, which is usually captured at high resolution to enable detection of even minute defects, at once, while simultaneously enabling efficient patch-level data processing in subsequent stages.

[0195] Additionally, in the embodiment, the computing system (1000) can extract features for each divided patch. (S205)

[0196] Specifically, the computing system (1000) can extract features for each of the k patches divided as above by working in conjunction with a predetermined pre-trained deep learning model.

[0197] In an example, the computing system (1000) can extract features for each patch of target vision data using a predetermined pre-trained deep learning model capable of extracting patch-level features, such as a model trained through a large dataset like ImageNet and / or a foundation model in which self-supervised representation learning based on a large dataset is performed.

[0198] More specifically, the computing system (1000) can input each patch divided as above into the pre-trained deep learning model to extract features for each patch of the target vision data in the form of a feature vector and / or output token that compresses the visual information of the corresponding patch.

[0199] FIG. 9 illustrates an example of a drawing for explaining a method for extracting patch-specific features according to one embodiment of the present disclosure.

[0200] For a specific example, referring to FIG. 9, a computing system (1000) can use a pre-trained model based on a Convolutional Neural Network (CNN) to match the different levels of feature maps extracted from each convolution layer of the model that receives each divided patch as input to the same size through an interpolation method.

[0201] And the computing system (1000) can extract features for each patch of the target vision data by combining the values ​​of each feature map at the same (x, y) location to extract a final feature vector.

[0202] Additionally, in the embodiment, the computing system (1000) can measure the distance between the extracted patch-specific features and the patch-specific features of the representative good product data. (S207)

[0203] That is, the computing system (1000) can calculate the distance between the two features by comparing the patch-specific features (hereinafter, target patch features) extracted from the target vision data and the patch-specific features (hereinafter, representative patch features) of the representative good product data.

[0204] Specifically, the computing system (1000) can obtain patch-specific features (i.e., representative patch features) of representative good product data in the same manner as steps S203 to S205 described above.

[0205] At this time, according to the embodiment, the computing system (1000) may perform patch splitting and feature extraction of representative good product data in advance, store the results in memory, and load and use them when necessary.

[0206] Additionally, the computing system (1000) can measure the distance between each target patch feature of the target vision data and each representative patch feature of the representative good product data.

[0207] Specifically, the computing system (1000) can measure the distance between each target patch feature and each representative patch feature corresponding to the same location based on a predetermined distance measurement method (e.g., Cosine Distance, Euclidean Distance, or Manhattan Distance, etc.).

[0208] Additionally, in the embodiment, the computing system (1000) can determine a suspected defective patch based on the distance between measured features. (S209)

[0209] Here, a patch suspected of being defective according to the embodiment may refer to a patch in which the distance between features measured as described above exceeds a preset threshold, and it is determined that a subsequent fine analysis step is required.

[0210] Specifically, the computing system (1000) can compare the distance between features measured as above with a preset threshold.

[0211] Here, for example, the above threshold may include a fixed threshold empirically set through expert experiments, an adaptive threshold dynamically determined based on the statistical distribution (e.g., mean and standard deviation) of feature distance values ​​measured from multiple good samples, and / or an optimal threshold learned through a machine learning model to best distinguish between good and defective product feature distance distributions.

[0212] Additionally, the computing system (1000) can determine that a target patch feature corresponding to the comparison result is a good patch if the distance between the corresponding features is less than or equal to a preset threshold.

[0213] On the other hand, the computing system (1000) can determine that the corresponding target patch feature is a suspected defective patch if the distance between the corresponding features in the comparison result exceeds a preset threshold.

[0214] And the computing system (1000) can apply the patch determined to be a suspected defective patch to the processing target of the subsequent S107 step.

[0215] At this time, the computing system (1000) can perform the above process of determining suspected defective patches individually for each patch.

[0216] Thus, the computing system (1000) can primarily detect at least one suspected defective patch from the target vision data.

[0217] In this way, in the embodiment, the computing system (1000) can maximize the processing speed and efficiency of the entire inspection process by primarily scanning the entire target vision data through systematic processes such as data alignment, division, feature extraction, and comparison, and roughly selecting areas suspected of being defective, thereby efficiently excluding the majority of good product patches that do not require precise analysis and compressing the targets for processing in the subsequent precise analysis step into a small number of candidates.

[0218] Returning to FIG. 4, a computing system (1000) according to one embodiment of the present disclosure may obtain a prompt support set based on a first detected suspected fault patch. (S107)

[0219] Here, the prompt support set according to the embodiment may refer to a set of N (N>=1) similar sample patch-labels that are provided as input to a microclassification model along with a suspected defect patch used as a query (i.e., a suspected defect patch determined through primary detection (Coarse detection)) and used as a reference standard for secondary detection of the suspected defect patch.

[0220] At this time, the similar sample patch-label set according to the embodiment may be a data set comprising N sample patches (i.e., similar patches) dynamically selected from among the data stored in the data pool as having a similarity greater than or equal to a preset standard with a suspected defective patch for which a precise judgment is to be performed, and a pair of correct labels matched to the sample patches.

[0221] In detail, in an embodiment, the computing system (1000) can detect N sample patch-label data (i.e., similar patch-label data) having a similarity greater than a preset standard (e.g., being included in the top N with the shortest distance between patch features) from a data pool built according to the aforementioned step S101, based on a first detected suspected defective patch (hereinafter, defective query patch).

[0222] That is, the computing system (1000) can detect the top N sample patch-label data that are determined to have a pattern similar to the pattern of the bad query patch as similar patch-label data.

[0223] And in the embodiment, the computing system (1000) can obtain a prompt support set for bad query patches including N similar patch-label data detected.

[0224] More specifically, the computing system (1000) can obtain a prompt support set for bad query patches by using a predetermined similarity search algorithm (e.g., Nearest Neighbor Search, Approximate Nearest Neighbor Search, etc.).

[0225] FIG. 10 illustrates an example of a drawing for explaining a method of obtaining a Prompt Support Set according to one embodiment of the present disclosure.

[0226] As an example, referring to FIG. 10, a computing system (1000) can measure the distance between a feature of a bad query patch (e.g., a feature vector) and a feature of all sample patches stored in a data pool (e.g., a feature vector) using a Nearest Neighbor Search algorithm.

[0227] At this time, the computing system (1000) can calculate the distance between the aforementioned features using various indicators such as cosine distance, Euclidean distance and / or Manhattan distance.

[0228] And the computing system (1000) can form a prompt support set by selecting N similar patch-label data in order of the smallest distance between the measured features.

[0229] At this time, in the embodiment, the computing system (1000) may repeat the process of obtaining the prompt support set described above for each of at least one bad query patch.

[0230] Thus, the computing system (1000) can obtain a prompt support set for each of at least one bad query patch.

[0231] Additionally, a computing system (1000) according to one embodiment of the present disclosure can secondarily detect a suspected defective patch based on an acquired prompt support set. (S109)

[0232] In detail, in an embodiment, the computing system (1000) can perform a second detection (Fine detection) to precisely determine a suspected defective patch (i.e., a defective query patch) within the target vision data that has been first detected (Coarse detection) by referring to similar samples (i.e., a prompt support set) in a pre-established data pool.

[0233] FIG. 11 illustrates an example of a drawing for explaining a method for secondarily detecting a suspected defective patch based on a prompt support set according to one embodiment of the present disclosure.

[0234] More specifically, referring to FIG. 11, in the embodiment, the computing system (1000) can input the prompt support set and bad query patch obtained as described above into a pre-trained microclassification model according to step S101.

[0235] Then, the microclassification model according to the embodiment can compare and analyze which type (e.g., normal variation or specific type of defect) a defective query patch is closest to by referring to N similar patches included in the input prompt support set and the corresponding correct labels (i.e., N similar patch-label data).

[0236] And the micro-classification model can output and provide the predicted correct answer label (e.g., good product or defective product, etc.) for the analysis result, i.e., the defective query patch, to the computing system (1000).

[0237] At this time, in the embodiment, the computing system (1000) can repeat the secondary detection process described above for each of at least one bad query patch.

[0238] Thus, in the embodiment, the computing system (1000) can derive at least one suspected defective patch that is more precisely determined based on a secondary detection for each of the at least one suspected defective patch (i.e., a defective query patch) primarily detected from the target vision data.

[0239] In this way, in the embodiment, the computing system (1000) dynamically finds a group of data samples similar to the first detected suspected defect patch (i.e., defect query patch) to generate a prompt support set, and by referencing this as a direct input to the model to perform a label determination (e.g., good product or defective product) for the corresponding defect query patch, thereby implementing a second detection that more precisely distinguishes between the difference between a fine deformation of a good product and an actual defect, which is prone to misjudging by existing general classification models, and ultimately maximizes inspection accuracy.

[0240] In addition, a computing system (1000) according to one embodiment of the present disclosure can provide a detection result. (S111)

[0241] In detail, in an embodiment, the computing system (1000) can manifest and provide the detection results performed as above in a predetermined manner.

[0242] In an example, the computing system (1000) can generate a perceptible notification signal indicating the quality status of an object to be inspected based on the aforementioned detection result, and provide it by outputting it through a user interface.

[0243] As a specific example, the computing system (1000) can generate and provide visualization information based on the aforementioned detection results (e.g., information visually displaying the location, type, and / or reliability score of a patch determined to be a final defect in the source of the target vision data) through a predetermined graphical user interface (e.g., a manufacturing line monitoring dashboard and / or a quality control system screen).

[0244] As another example, the computing system (1000) can generate and provide audio information based on the aforementioned detection results (e.g., alarm and / or voice guidance such as “Line 3, scratch defect occurred”) through a predetermined audio user interface (e.g., field speaker and / or worker personal terminal, etc.).

[0245] In another embodiment, the computing system (1000) can transmit the results of the aforementioned detection to an external device and / or system to support the implementation of various services based on the results of the secondary detection.

[0246] As a specific example, the computing system (1000) records the aforementioned detection results in a predetermined database (e.g., a SQL (Structured Query Language) database and / or a time-series database, etc.) and transmits them to an external device and / or system, thereby supporting the implementation of various quality control and data analysis services (e.g., manufacturing process quality inspection, product history tracking, defect type statistical analysis, logistics and distribution management, medical image analysis, smart factory, etc.) based on the secondary detection results, and can effectively support the operation of various advanced services and / or data-based decision-making, such as tracking the production history of a product, managing real-time yield, or statistically analyzing the occurrence trends of specific defect types.

[0247] In another embodiment, the computing system (1000) can transmit the result of the aforementioned detection to an external device and / or system to trigger a certain automated action corresponding to the result of the secondary detection.

[0248] As a specific example, the computing system (1000) can operate as an AI agent based on the aforementioned detection result and trigger an automated follow-up action based on the detection result.

[0249] For example, the computing system (1000) can transmit a control signal to a sorting device on a manufacturing line that produces a product determined to be a ‘defective product (NG)’ to separate the product from the manufacturing line, or automatically transmit a quality inspection result (i.e., a detection result) to a manufacturing execution system (MES) to update the production history in real time.

[0250] In this way, in the embodiment, the computing system (1000) applies a Coarse-to-Fine approach in which it rapidly selects a suspected defective area by comparing features with a good product (Coarse stage), and then dynamically finds the most similar samples from a data pool to configure a prompt for that area to make a precise second judgment (Fine stage), thereby effectively resolving the trade-off between processing speed and judgment accuracy, which was a limitation of the existing inspection method, and effectively implementing high-speed and high-precision vision inspection.

[0251] Meanwhile, a computing system (1000) operating according to a Coarse-to-Fine approach according to one embodiment of the present disclosure as described above may have technical significance in various aspects.

[0252] For example, from the perspective of an intelligent agent (AI Agent), the computing system (1000) can perform step-by-step decision-making, such as operating as an intelligent agent (AI Agent) to perform a simple scan for searching suspicious areas (Coarse stage) and calling a detailed module (i.e., a fine classification model) for precise classification based on the result.

[0253] Through this, the computing system (1000) can implement an intelligent judgment structure that satisfies both inspection accuracy and speed by employing a multi-step strategy.

[0254] As another example, from the perspective of a Chain of Experts, a computing system (1000) can implement a structure in which modules with different expertise collaborate sequentially.

[0255] Specifically, a method can be implemented in which components with various expertise cooperate sequentially to efficiently solve complex inspection problems, such as acting as suspicious patch detection experts in the first detection (Coarse) stage and acting as similarity judgment experts for data pool search and fine classification experts for final judgment in the second detection (Fine) stage.

[0256] As another example, in terms of the Chain of Thought, the computing system (1000) can perform step-by-step reasoning similar to the human thought process.

[0257] Specifically, the computing system (1000) can first perform a first inference (Coarse stage) to roughly examine the entire area to detect a suspected area, and then perform a second inference (Fine stage) to definitively classify only the suspected area by taking more detailed information from the data pool, and this dual thinking process can effectively implement a step-by-step inference method.

[0258] As another example, from the perspective of a SmartFactory, a computing system (1000) can function as a key element for the intelligentization of the manufacturing process.

[0259] Specifically, the computing system (1000) can maximize the efficiency of real-time monitoring and quality inspection, such as rapidly processing large volumes of high-resolution vision data at the production line site and precisely analyzing only the suspected area while omitting detailed inspection of the entire unnecessary area.

[0260] In conclusion, the computing system (1000) can provide an advanced vision inspection solution that satisfies multi-stage decision-making from the perspective of an AI Agent, expert module collaboration from the perspective of a Chain of Expert, step-by-step reasoning from the perspective of a Chain of Thought, and real-time high-precision requirements from the perspective of a Smart Factory, through a step-by-step approach based on a Coarse-to-Fine Framework according to one embodiment of the present disclosure.

[0261]

[0262] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present disclosure, and vice versa.

[0263] The specific embodiments described in this disclosure are examples and do not limit the scope of this disclosure in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of this disclosure.

[0264] Furthermore, although the detailed description of the present disclosure has been explained with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that the present disclosure can be modified and changed in various ways without departing from the spirit and technical scope of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0265]

[0266] The present disclosure relates to a vision inspection method and system based on coarse-to-fine patch-level classification, and since it is applicable to the artificial intelligence industry, it has industrial applicability.

Claims

1. In a method executed by a computer, A step in which at least one processor of the computer accesses a data structure of at least one memory of the computer; - wherein the data structure includes target vision data which is data specifying an object to be inspected, and The step of the above at least one processor loading the target vision data from the above at least one memory; The above at least one processor extracts patch-specific features of the loaded target vision data; The step of the above at least one processor detecting at least one suspected defective patch within the target vision data based on the extracted patch-specific features; The above-mentioned at least one processor acquires a Prompt Support Set based on the first detected suspected defect patch; - wherein the Prompt Support Set includes at least one sample patch having a similarity greater than or equal to a preset standard with the first detected suspected defect patch and a correct answer label matching the sample patch, The step of the at least one processor inputting the acquired prompt support set and the first detected suspected defect patch into at least one artificial intelligence model to secondarily detect at least one suspected defect patch within the target vision data; and A method comprising the step of at least one processor manifesting the result of the secondary detection through an interface.

2. In Paragraph 1, The step of extracting patch-specific features of the above target vision data is: A step of estimating the geometric transformation relationship between the pre-set good product vision data and the target vision data, and A method comprising the step of aligning the target vision data according to the coordinate system of the good product vision data based on the estimated geometric transformation relationship.

3. In Paragraph 2, The step of extracting patch-specific features of the above target vision data is: The step of dividing the above-mentioned aligned target vision data into patches of a certain shape, and A method further comprising the step of extracting features for each of the above-described divided patches.

4. In Paragraph 3, The step of primarily detecting the above-mentioned suspected defective patch is, A step of measuring the distance between the extracted patch-specific features and the patch-specific features of the good product vision data, and A method comprising the step of primarily determining the suspected defective patch based on the distance between the measured features.

5. In Paragraph 1, A method further comprising the steps of: the at least one processor collecting at least one vision data; dividing the collected vision data into patches of a certain shape; assigning a correct label to each of the at least one divided patches; and building a data pool by matching and storing the assigned correct label and the patch corresponding to the correct label.

6. In Paragraph 5, The above correct answer label is, A method defined as a type specifying a good product type, a defective product type, or a specific shape of a defect.

7. In Paragraph 5, The step of acquiring the above prompt support set is, A method comprising the step of detecting, among the at least one patch included in the constructed data pool, at least one patch having a similarity greater than or equal to a preset standard with the first detected suspected defective patch, and a correct answer label matching the at least one patch having a similarity greater than or equal to the preset standard.

8. In Paragraph 7, The step of acquiring the above prompt support set is, A step of measuring the distance between the features of the first detected suspected defective patch and the features of the at least one patch included in the data pool based on a predetermined similarity search algorithm, and Among the at least one patch included in the data pool, the step of selecting at least one patch in order of smallest distance between the measured features, and A method further comprising the step of obtaining the prompt support set by detecting at least one selected patch and a correct answer label matching the at least one selected patch.

9. In Paragraph 5, The above at least one processor further includes the step of training the at least one artificial intelligence model, - wherein the at least one artificial intelligence model is an artificial intelligence model pre-trained to predict a correct label for a query patch to be examined based on at least one sample patch to be referenced and a correct label matching the sample patch, and The step of training at least one artificial intelligence model is, The step of randomly extracting at least one patch and single patch data to which the correct answer label is assigned from the above data pool, and The step of generating a training episode dataset based on at least one randomly extracted patch and single patch data, and A method comprising the step of training at least one artificial intelligence model to predict a correct label for a single patch data by referring to at least one patch to which the correct label is assigned, based on the above-mentioned generated training episode dataset.

10. In Paragraph 9, The step of secondarily detecting the above-mentioned suspected defective patch is, A method comprising the step of inputting the obtained prompt support set into the at least one artificial intelligence model as a reference target, and inputting the first detected suspected defective patch into the at least one artificial intelligence model as an inspection target.

11. In Paragraph 1, The step of secondarily detecting the above-mentioned suspected defective patch is, A step of predicting a correct label for a first-detected suspected defect patch based on at least one sample patch having a similarity greater than or equal to a preset standard with the first-detected suspected defect patch included in the prompt support set, and a correct label matching the sample patch; A method comprising the step of outputting the result of the second detection for the first detected suspected defective patch based on the predicted correct answer label.

12. In Paragraph 1, The step of implementing the result of the above secondary detection through an interface is, A step of generating a perceptible notification signal indicating the quality status of the object under inspection based on the result of the second detection above, and A method comprising the step of providing the generated recognizable notification signal through the interface.

13. In Paragraph 1, The step of implementing the result of the above secondary detection through an interface is, A method comprising the step of transmitting the result of the second detection to at least one of an external device and a system to implement a service based on the result of the second detection.

14. In Paragraph 1, The step of implementing the result of the above secondary detection through an interface is, A method comprising the step of transmitting the result of the second detection to at least one of an external device and system to trigger a predetermined automated measure corresponding to the result of the second detection.

15. At least one processor; and It includes at least one memory that stores at least one instruction that performs the following when executed by the above at least one processor; and The above at least one instruction is, The above-mentioned at least one processor has the step of accessing a data structure of the above-mentioned at least one memory, - wherein the data structure includes target vision data which is data specifying an object to be inspected, and The above at least one processor, the step of loading the target vision data from the above at least one memory, and The above at least one processor extracts patch-specific features of the loaded target vision data, and The above-mentioned at least one processor performs a step of primarily detecting at least one suspected defective patch within the target vision data based on the extracted patch-specific features, and The above-mentioned at least one processor acquires a Prompt Support Set based on the first detected suspected defect patch, - wherein the Prompt Support Set includes at least one sample patch having a similarity greater than or equal to a preset standard with the first detected suspected defect patch and a correct answer label matching the sample patch, The above at least one processor inputs the acquired prompt support set and the first detected suspected defect patch into at least one artificial intelligence model to secondarily detect at least one suspected defect patch within the target vision data, and A system comprising instructions in which at least one processor performs the step of manifesting the result of the secondary detection through an interface.

16. In Paragraph 15, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; A system further comprising a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

17. In Paragraph 15, A plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; comprising A system further comprising an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through any one of the plurality of synaptic circuits.

Images