Electronic device and method for detecting defect of product

Abstract

An electronic device that detects a defect of a product, according to an embodiment of the present invention, may comprise a processor that receives first time series data generated according to a process of manufacturing a first product, extracts at least one process factor of the first time series data and a plurality of process values corresponding to the at least one process factor, generates first image data represented by a heat map on the basis of color values corresponding to the respective process values, and detects whether the first product is defective, by inputting the first image data to an image-based model.

Description

Electronic device and method for detecting product defects

[0001] The present invention relates to an electronic device and method for detecting product defects.

[0002] Recently, with the development of related technologies such as the Internet of Things (IoT) and Artificial Intelligence (AI), smart factories have become a hot topic. A smart factory is a general term for a manufacturing system operated by applying Information and Communications Technology (ICT) combined with digital automation solutions to production processes such as design, development, and manufacturing.

[0003] In workplaces where smart factory technology is applied, labor costs can be reduced as the deployment of workers is not required. Additionally, since multiple processes for product manufacturing are performed continuously, production speed and unit costs can be reduced. As a result, active attempts are being made to apply smart factories in various field situations.

[0004] One example is the increasing number of attempts to utilize AI models to detect product defects following the commercialization of AI models. Most AI models currently in use rely on time-series data, which presents a limitation in that it is difficult to identify the relationships between the large volumes of data used for analysis. Furthermore, being restricted to time-series data not only limits the available models but also leads to resource issues, such as memory limitations, when processing data as the length of numerical data increases.

[0005] Therefore, in addition to utilizing time-series data, there is a need to devise a method to detect product defects more accurately while reducing resource usage.

[0006] The objective of the present invention is to provide an electronic device and method for detecting product defects more accurately and efficiently.

[0007] To solve the above technical problem, an electronic device for detecting defects in a product according to one embodiment of the present invention comprises: a memory for storing at least one instruction; and a processor for executing said instruction. The processor receives time-series data according to a process for manufacturing a product, extracts at least one process factor and a plurality of process values ​​corresponding to said at least one process factor from said time-series data, generates image data based on said at least one process factor and said plurality of process values, and inputs said image data into an image-based model to detect whether the product is defective.

[0008] In one embodiment of the present invention, the processor can generate the image data represented as a heatmap based on a color value corresponding to the process value.

[0009] In one embodiment of the present invention, the processor determines the at least one process factor and the reference color constituting the image data based on the image-based model, and can identify a color value corresponding to each process value using the reference color.

[0010] In one embodiment of the present invention, the processor can be used to generate the image data by normalizing the plurality of process values.

[0011] In one embodiment of the present invention, the processor can identify a color value corresponding to each process value according to a reference color determined based on a data range for each process factor.

[0012] In one embodiment of the present invention, the image-based model may be trained to indicate at least one region in the image data that influenced the determination that the product is defective.

[0013] In one embodiment of the present invention, the processor can identify process information regarding defects based on at least one area of ​​the image data.

[0014] In one embodiment of the present invention, the processor can generate at least one graph based on at least one process factor and the plurality of process values, and convert the at least one graph into image data.

[0015] In one embodiment of the present invention, the processor can determine the at least one process factor and a variable corresponding to the axis of the graph based on the image-based model.

[0016] In one embodiment of the present invention, when a plurality of graphs are generated, the processor can convert the plurality of graphs into a single image data.

[0017] In addition, to solve the above technical problem, a method for detecting defects in a product according to an embodiment of the present invention comprises: receiving time series data according to a process for manufacturing a product; extracting at least one process factor and a plurality of process values ​​corresponding to the at least one process factor from the time series data; generating image data based on the at least one process factor and the plurality of process values; and

[0018] It may include a step of detecting whether the product is defective by inputting the image data into an image-based model.

[0019] In one embodiment of the present invention, the image data represented as a heatmap can be generated based on color values ​​corresponding to the process values.

[0020] In one embodiment of the present invention, the method may further include the step of determining the reference color constituting the at least one process factor and the image data based on the image-based model; and the step of identifying a color value corresponding to each process value using the reference color.

[0021] In one embodiment of the present invention, the step of normalizing the plurality of process values ​​and using them to generate the image data may be further included.

[0022] In one embodiment of the present invention, the method may further include the step of identifying a color value corresponding to each process value according to a reference color determined based on a data range for each process factor.

[0023] In one embodiment of the present invention, the image-based model may be trained to indicate at least one region in the image data that influenced the determination that the product is defective.

[0024] In one embodiment of the present invention, the step of identifying process information regarding defects based on at least one region of the image data may be further included.

[0025] In one embodiment of the present invention, the method may further include the step of generating at least one graph based on at least one process factor and the plurality of process values; and the step of converting the at least one graph into image data.

[0026] In one embodiment of the present invention, the method may further include the step of determining the at least one process factor and a variable corresponding to the axis of the graph based on the image-based model.

[0027] In one embodiment of the present invention, when a plurality of graphs are generated, the step of converting the plurality of graphs into a single image data may be further included.

[0028] According to one embodiment of the present invention, data generated in manufacturing, such as PLCs, is often converted into a numerical form by a data memory map, enabling fast processing and accurate analysis.

[0029] According to one embodiment of the present invention, image data contains information spatially (2D or 3D), so that rich information can be provided at a single data point.

[0030] According to one embodiment of the present invention, image data contains information spatially (2D or 3D), so that various data can be reflected on a single image.

[0031] FIG. 1 is a schematic diagram illustrating a manufacturing system according to one embodiment of the present invention.

[0032] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0033] FIG. 3 is a diagram illustrating the operation flowchart of an electronic device according to one embodiment of the present invention.

[0034] FIG. 4 is a drawing illustrating image data according to the first embodiment of the present invention.

[0035] FIG. 5 is a drawing illustrating image data according to a second embodiment of the present invention.

[0036] FIG. 6 is a drawing illustrating image data according to a third embodiment of the present invention.

[0037] FIG. 7 is a drawing illustrating the operation of an electronic device according to one embodiment of the present invention.

[0038] FIG. 8 is a diagram illustrating the operation flowchart of an electronic device according to one embodiment of the present invention.

[0039] FIG. 9 is a drawing illustrating image data according to the fourth embodiment of the present invention.

[0040] FIG. 10 is a drawing illustrating the operation of an electronic device according to one embodiment of the present invention.

[0041] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiment in which the present invention can be practiced. In order to clearly explain the present invention in the drawings, parts unrelated to the description may be omitted, and the same reference numerals may be used for identical or similar components throughout the specification.

[0042] FIG. 1 is a schematic diagram illustrating a manufacturing system according to one embodiment of the present invention.

[0043] Referring to FIG. 1, the manufacturing system (1) (hereinafter also referred to as the system (1)) may include an electronic device (100), a server (200), a power management device (310) for each process line (300), an equipment control device (320), and an equipment module (330).

[0044] An electronic device (100) according to one embodiment of the present invention is a device that detects product defects by converting time-series data collected from a process line (300) into image data, and can be implemented as a computer, PLC (Programmable Logic Controller), server, smartphone, tablet PC, smart pad, laptop, etc. The electronic device (100) can embed a Manufacturing Execution System (MES), an Employee Assistance Program (EAP), a Recipe Parameter Management System (RPMS), etc.

[0045] A server (200) according to one embodiment of the present invention communicates with an electronic device (100), a device on a process line (300), etc., and is a device that collects or provides data, and can embed a Manufacturing Execution System (MES), an Employee Assistance Program (EAP), a Recipe Parameter Management System (RPMS), etc.

[0046] A server (200) according to one embodiment of the present invention may be implemented as a separate device from the electronic device (100), but is not limited thereto and may also be implemented as a single device. In addition, the electronic device (100) and the server (200) may not be physically separated but may be implemented in various ways, such as by being controlled separately.

[0047] A process line (300) (process line #1, process line #2 ...) according to one embodiment of the present invention may include a plurality of equipment modules (330) configured to operate sequentially according to the process order, and at least one process line (300) may be installed depending on the site conditions.

[0048] A plurality of equipment modules (330) (equipment module #1, equipment module #2, equipment module #3, ...) may be connected to an equipment control device (320) (equipment control device #1, equipment control device #2, equipment control device #3, ...) that controls the operation of each equipment module. Additionally, the process line (300) may separately be equipped with a power management device (310) that manages the power supply, etc., of the plurality of equipment control devices (320).

[0049] At this time, the power management device (310) and the equipment control device (320) can be implemented as a PLC (Programmable Logic Controller), but are not limited thereto, and can be implemented as a computer, tablet PC, smart pad, laptop, etc.

[0050] As described above, in order to overcome the limitations of product defect detection using time series data, the present invention proposes a method for detecting product defects by converting time series data into image data and using an image-based model.

[0051] Hereinafter, the configuration and operation of a manufacturing system (1) according to one embodiment of the present invention will be described in detail with reference to the drawings.

[0052] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0053] An electronic device (100) according to one embodiment of the present invention includes an input unit (110), a communication unit (120), a display unit (130), a storage unit (140), and a processor (150).

[0054] The input unit (110) generates input data in response to user input of the electronic device (100). For example, the user input may be a user input that operates the electronic device (100), a user input that sets a data range for each process factor, a user input that sets a reference color corresponding to the data range, etc. In addition to this, any user input necessary to detect product defects can be applied without limitation.

[0055] Alternatively, user input may be user input that sets variables corresponding to the axes of the graph.

[0056] The input unit (110) includes at least one input means. The input unit (110) may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a mouse, a menu button, etc.

[0057] The communication unit (120) can communicate with external devices such as a server (200), a power management device (310), an equipment control device (320), and an equipment module (330) to transmit and receive time-series data, image data, image-based models, etc.

[0058] To this end, the communication unit (120) can perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Wi-Fi (wireless fidelity), Bluetooth, or wired communication such as LAN (local area network), WAN (Wide Area Network), and power line communication.

[0059] The display unit (130) displays display data according to the operation of the electronic device (100). The display unit (130) includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (130) can be combined with the input unit (110) to be implemented as a touch screen.

[0060] The storage unit (140) stores operation programs of the electronic device (100). The storage unit (140) includes storage with non-volatile properties that can preserve data (information) regardless of whether power is provided, and memory with volatile properties in which data to be processed by the processor (150) is loaded and data cannot be preserved if power is not provided. Storage includes flash memory, hard-disc drive (HDD), solid-state drive (SSD), and ROM (Read Only Memory), and memory includes buffer and RAM (Random Access Memory).

[0061] The storage unit (140) can store a Manufacturing Execution System (MES), time series data, image data, image-based models, etc. The storage unit (140) can store computation programs, etc., that are necessary in the process of receiving user input, extracting process factors and process values, generating image data, identifying color values, and detecting whether a product is defective.

[0062] The storage unit (140) may store a computation program required during the process of generating a graph.

[0063] The processor (150) can execute software, such as a program, to control at least one other component (e.g., hardware or software component) of the electronic device (100) and can perform various data processing or operations.

[0064] A processor (150) according to one embodiment of the present invention receives first time series data generated according to a process for manufacturing a first product, extracts at least one process factor and a plurality of process values ​​corresponding to the at least one process factor among the first time series data, generates first image data represented as a heatmap based on color values ​​corresponding to each process value, and inputs the first image data into an image-based model to detect whether the first product is defective.

[0065] Additionally, the processor (150) receives first time series data according to the process of manufacturing the first product, extracts at least one process factor and a plurality of process values ​​corresponding to the at least one process factor from the first time series data, generates at least one graph based on the at least one process factor and the plurality of process values, converts the at least one graph into first image data, and inputs the first image data into an image-based model to detect whether the first product is defective.

[0066] At this time, the processor (150) may train an image-based model trained to detect whether the product is defective, or receive and store an image model that has been trained and built from the outside and use it, and is not limited to either one.

[0067] Meanwhile, the processor (150) may perform at least some of the data analysis, processing, and result information generation for performing the above operations using at least one of machine learning, neural network, or deep learning algorithms as a rule-based or artificial intelligence algorithm. Examples of neural networks may include models such as CNN (Convolutional Neural Network), DNN (Deep Neural Network), and RNN (Recurrent Neural Network).

[0068] FIG. 3 is a diagram illustrating the operation flowchart of an electronic device according to one embodiment of the present invention.

[0069] According to one embodiment of the present invention, the processor (150) can receive first time series data according to the process of manufacturing the first product (S10).

[0070] Time series data refers to data generated from each equipment module according to the manufacturing process, acquired over time. Time series data can consist of process values ​​acquired over time for each process factor.

[0071] Time series data may be, for example, data received from a sensor attached to the equipment module (330). The sensor may include, for example, a current sensor, a voltage sensor, a temperature sensor, a vibration sensor, a rotational speed / position sensor, etc. In addition, the time series data may be data received through an equipment control device (320) or a server (200) that controls the equipment module (330). The time series data may be all device (memory) input / output data within a program of a production process site using a PLC.

[0072] Time series data is data that can be received from all equipment control devices (320) and / or equipment modules (330), and the types, number, and receiving paths of process factors included in the time series data are not limited to any one.

[0073] According to one embodiment of the present invention, the processor (150) can extract at least one process factor and a plurality of process values ​​corresponding to at least one process factor from the first time series data (S20).

[0074] The processor (150) can determine at least one process factor based on an image-based model. The image-based model is an artificial intelligence model trained to determine whether a product is defective based on image data, and may include, but is not limited to, various types of artificial intelligence models such as CNN (Convolutional Neural Network), YOLO, Resnet, AlexNet, DNN (Deep Neural Network), RNN (Recurrent Neural Network), R-CNN (Region with Convolutional Neural Network), S-DNN (Stacking-based Deep Neural Network), RPN (Region Proposal Network), DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory), and Transformer.

[0075] Image-based models can be created for each type of image data, or they can be trained based on two or more types of image data. For example, an image-based model can be trained to determine whether a product is defective using image data obtained by converting current data (time-series data) into an image. As another example, an image-based model can be trained to determine whether a product is defective using image data obtained by converting current data and voltage data (time-series data) into an image. In this case, the image data may be obtained by converting current data and voltage data (time-series data) into image data separately, or it may be obtained by combining the time-series data and converting them into a single image data.

[0076] As such, the process factors used to detect product defects may differ from model to model in the image-based model. Accordingly, the processor (150) can determine at least one process factor based on the image-based model.

[0077] According to one embodiment of the present invention, the processor (150) can generate first image data based on a color value corresponding to a process value (S30).

[0078] The first image data can be represented in the form of a heatmap, in which time-series process values ​​are visualized as colors. Examples of this are illustrated in FIGS. 4 to 7.

[0079] Just as the learned process factors differ for each image-based model, the reference color representing the process value can be set differently for each image-based model. Accordingly, the processor (150) can determine the reference color that constitutes the image data based on the image-based model.

[0080] A reference color refers to two or more colors. For example, if the reference colors are white and red, each process value can be determined between the RGB reference white color value (255, 255, 255) and the red color value (255, 0, 0). Color values ​​can be represented in various formats, such as HEX (Hexadecimal) and Lab (CIE-Lab), in addition to RGB, and are not limited to any one of them. For the sake of convenience, the following explanation will be based on RGB.

[0081] The processor (150) can identify a color value corresponding to each process value using a determined reference color. At this time, the reference color can be determined based on the data range for each process factor. For example, the minimum value of the current data range can be determined as red, the middle value as yellow, and the maximum value as green as the reference color.

[0082] Additionally, the processor (150) can normalize multiple process values. Normalizing multiple process values ​​means adjusting process values ​​that appear as numbers in various ranges to a certain standard, and various normalization methods may be adopted.

[0083] For example, Min-Max Scaling can be applied to normalize the range of data to values ​​between 0 and 1. This can be expressed mathematically as follows.

[0084]

[0085] x is the original process value, x min is the minimum value in the data, x max is the maximum value in the data, x norm means a normalized value.

[0086] As another example, Z-score normalization can be applied by subtracting the mean and dividing by the standard deviation. This can be expressed mathematically as follows.

[0087]

[0088] μ is the mean of the data, and σ is the standard deviation.

[0089] For example, if a plurality of process values ​​are normalized to values ​​between 0 and 1 based on the above methods, the white color value (255, 255, 255) corresponds to 0, and the red color value (255, 0, 0) corresponds to 1. Based on this, the processor (150) can identify the color value corresponding to each process value existing between 0 and 1.

[0090] When process data is normalized in this way, comparison and analysis of the data become easier, and consistent color values ​​can be displayed when generating a heatmap based on the normalized values. Therefore, the processor (150) can generate image data by identifying each value as a color value based on the range of the process data. This can help the image-based model determine whether there is a defect more accurately.

[0091] According to one embodiment of the present invention, the processor (150) can detect whether the first product is defective by inputting the first image data into an image-based model (S40).

[0092] The output of the image-based model may be whether the first product is good or defective.

[0093] An image-based model can be trained in various ways; for example, it can be trained using multiple image data indicating a good product and multiple image data indicating a defective product as training data. In this case, image data generated based on at least one process factor affecting whether the product is good or not can be used as training data.

[0094] Meanwhile, an image-based model can be trained to indicate at least one region in the image data that influenced the determination that the product is defective. This is described in detail with reference to Fig. 7.

[0095] According to one embodiment of the present invention, data generated in manufacturing, such as PLCs, is often converted into a numerical form by a data memory map, enabling fast processing and accurate analysis.

[0096] According to one embodiment of the present invention, image data contains information spatially (2D or 3D), so that rich information can be provided at a single data point.

[0097] According to one embodiment of the present invention, there is a possibility of intuitive interpretation through image data, and it is easy to visually interpret or verify the results learned by the model.

[0098] According to one embodiment of the present invention, it can be utilized in various fields such as manufacturing as well as autonomous driving, thus offering excellent versatility.

[0099] According to one embodiment of the present invention, image data usually has high dimensions, so it is suitable for learning complex patterns.

[0100] According to one embodiment of the present invention, in the field of image processing, there are abundant pre-trained models, so high performance can be obtained even with a small amount of data.

[0101] FIG. 4 is a drawing illustrating image data according to the first embodiment of the present invention.

[0102] FIG. 4 illustrates image data generated using process values ​​of process factors (coating amount, process time, thickness) in a coating process. At this time, it is assumed that image data (410) is image data when the product is good, and image data (420) is image data when the product is defective.

[0103] Image data (410) and image data (420) represent the coating amount (①), process time (②), and thickness (③), which are process factors in the coating process, and the process values ​​corresponding to each process factor in color.

[0104] At this time, the processor (150) can identify a color value corresponding to each process value using the determined reference color. At this time, the reference color can be determined based on the data range for each process factor. For example, the minimum value of each data range can be determined as red, the middle value as yellow, and the maximum value as green as the reference color.

[0105] The processor (150) can input image data (410) and image data (420) into an image-based model to obtain good and defective results, respectively.

[0106] At this time, due to its characteristics, the image data (420) can intuitively and visually identify points that are distinguishable from normal data. For example, the area (421) and area (422) of the image data (420) may differ from the corresponding parts of the image data (410). Therefore, the operator will be able to confirm that a defect has occurred due to the process time (②) and thickness (③).

[0107] FIG. 5 is a drawing illustrating image data according to a second embodiment of the present invention.

[0108] FIG. 5 shows the image data (420) of FIG. 4 modified using two reference colors, and can be visualized using two reference colors, green and white. As such, the reference colors may be two or more colors, and the number of reference colors can be applied without limitation.

[0109] FIG. 6 is a drawing illustrating image data according to a third embodiment of the present invention.

[0110] FIG. 6 is an image of time-series data by process, and the image data (600) can display the process values ​​of process #1 (①), process #2 (②), process #3 (③), process #4 (④), and process #5 (⑤) in different colors. At this time, the process factors may be the same or different.

[0111] According to one embodiment of the present invention, values ​​for measurements for each process can be separated and distinguished by color, making it easy to visually distinguish them.

[0112] FIG. 7 is a drawing illustrating the operation of an electronic device according to one embodiment of the present invention.

[0113] As previously described in relation to S40 in Fig. 3, an image-based model can be trained to indicate at least one area in the image data that influenced the determination that the product is defective.

[0114] A representative method for identifying at least one region that influenced the determination that a product is defective is to use the Class Activation Map (CAM) technique.

[0115] The Class Activation Map technique is a visualization method used in deep learning, particularly in CNNs, that visually indicates which parts of an input image are important when a model makes a prediction. Through this technique, the model's decision-making process can be interpreted. The Class Activation Map generates feature maps for each class using a Global Average Pooling layer before being passed to the final fully connected layer. The feature maps calculate a weighted sum based on class weights and process this using a ReLU function to generate a heatmap that highlights the important areas for the corresponding class. In this case, the heatmap is distinct from the image data itself; in FIG. 7, important areas (710, 720, 730) that influenced the determination that the image data (700) is defective are depicted in the form of a heatmap. In this case, the higher the weight for an area (the greater the influence), the darker the color is displayed.

[0116] It goes without saying that various techniques, such as Saliency Maps and Occlusion Sensitivity, can be applied in addition to the Class Activation Map technique.

[0117] The processor (150) can identify process information regarding a defect based on at least one first region of the first image data. At this time, the process information may include at least one of the process name where the defect occurred, the equipment module, the time of occurrence, the process factor, and the process value.

[0118] In addition, process information may include a product ID or serial number that can uniquely identify a specific product; information on the worker who performed the process (e.g., worker ID, name, shift, etc.); a unique ID of the production line where the defect occurred; equipment status or maintenance records at the time the process was performed; the environment in which the process was performed (e.g., temperature, humidity, cleanliness, etc.); information on the raw materials used in the process, such as the type of raw material, batch number, and supplier; the process step where the defect occurred and its position within the overall process sequence; inspection equipment information, such as the type and settings of the inspection equipment that detected the defect; setting values ​​of parameters used when performing the process (e.g., temperature settings, pressure settings, etc.); process information and inspection results before and after the process; the date and time when production started and ended; the lot number of the relevant production batch; information on energy consumption (electricity, gas, etc.) used during the process; and production speed and cycle time.

[0119] According to one embodiment of the present invention, by identifying the cause of the defect, it may be easy to devise a countermeasure.

[0120] FIG. 8 is a diagram illustrating the operation flowchart of an electronic device according to one embodiment of the present invention.

[0121] According to one embodiment of the present invention, the processor (150) can receive second time series data according to the process of manufacturing the second product (S11).

[0122] Time series data refers to data generated from each equipment module according to the manufacturing process, acquired over time. Time series data can consist of process values ​​acquired over time for each process factor.

[0123] Time series data may be, for example, data received from a sensor attached to the equipment module (330). The sensor may include, for example, a current sensor, a voltage sensor, a temperature sensor, a vibration sensor, a rotational speed / position sensor, etc. In addition, the time series data may be data received through an equipment control device (320) or a server (200) that controls the equipment module (330). The time series data may be all device (memory) input / output data within a program of a production process site using a PLC.

[0124] Time series data is data that can be received from all equipment control devices (320) and / or equipment modules (330), and the types, number, and receiving paths of process factors included in the time series data are not limited to any one.

[0125] According to one embodiment of the present invention, the processor (150) can extract at least one process factor and a plurality of process values ​​corresponding to at least one process factor from the second time series data (S21).

[0126] The processor (150) can determine at least one process factor based on an image-based model. The image-based model is an artificial intelligence model trained to determine whether a product is defective based on image data, and may include, but is not limited to, various types of artificial intelligence models such as CNN (Convolutional Neural Network), YOLO, Resnet, AlexNet, DNN (Deep Neural Network), RNN (Recurrent Neural Network), R-CNN (Region with Convolutional Neural Network), S-DNN (Stacking-based Deep Neural Network), RPN (Region Proposal Network), DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory), and Transformer.

[0127] Image-based models can be created for each type of image data, or they can be trained based on two or more types of image data. For example, an image-based model can be trained to determine whether a product is defective using image data obtained by converting current data (time-series data) into an image. As another example, an image-based model can be trained to determine whether a product is defective using image data obtained by converting current data and voltage data (time-series data) into an image. In this case, the image data may be obtained by converting current data and voltage data (time-series data) into image data separately, or it may be obtained by combining the time-series data and converting them into a single image data.

[0128] As such, the process factors used to detect product defects may differ from model to model in the image-based model. Accordingly, the processor (150) can determine at least one process factor based on the image-based model.

[0129] According to one embodiment of the present invention, the processor (150) can generate at least one graph based on at least one process factor and a plurality of process values ​​(S31).

[0130] Just as the learned process factors differ for each image-based model, the variables corresponding to the axes of the graph can be set differently for each image-based model. Accordingly, the processor (150) can determine the variables corresponding to the axes of the graph based on the image-based model.

[0131] The variables corresponding to the axes of the graph may, for example, be time on the x-axis and process factors on the y-axis. As another example, the x-axis may be process factor #1 and the y-axis may be process factor #2. The graph is not limited to 2 dimensions and may be composed of n dimensions (where n is a natural number greater than or equal to 2). The method of generating the graph may utilize previously disclosed techniques and is not limited to any one of them.

[0132] Meanwhile, if the x-axis is set to time and the y-axis to process factors, this setting can visually show how the process factors change over time.

[0133] In particular, it allows one to understand how process factors change over time, making it useful for identifying long-term trends and patterns. It also enables easy identification of outliers or sudden changes occurring at specific points in time. This information assists in diagnosing the causes of problems and taking corrective actions. Furthermore, visualizing and analyzing process factors facilitates preventive maintenance and process optimization. By examining the correlation between two factors, one can understand the impact of a change in one factor on another within the process. This allows for the early detection of problems or the identification of efficient improvement methods.

[0134] In addition, when the x-axis is set to process factor #1 and the y-axis to process factor #2, this setting can visually show how the two process factors are correlated with each other.

[0135] When there are periodic changes in a process, the patterns can be visually identified to enable appropriate responses. By plotting process factors on the x-axis and y-axis, the relationship between the two factors can be intuitively visualized. This allows for a clear understanding of correlations and the identification of optimization opportunities. In this case, multivariate analysis is also possible, enabling the understanding of more complex patterns by analyzing multiple factors simultaneously. Such analysis can be effectively utilized to improve processes, resolve issues, and plan preventive maintenance schedules.

[0136] As such, the selection of variables corresponding to the axes of the graph varies depending on the nature of the data to be visualized and the purpose of the analysis, and each setting can provide valid information suitable for that purpose.

[0137] According to one embodiment of the present invention, the processor (150) can convert at least one graph into second image data (S41).

[0138] In the present invention, the graph itself is converted into image data and modeled. At this time, if multiple graphs are generated, the processor (150) can convert the multiple graphs into a single image data. Whether the process factors for each of the multiple graphs are identical will not have an effect.

[0139] There are various approaches to converting multiple graphs into a single image, as follows.

[0140] First, graphs can be overlaid. Multiple data points can be displayed on a single graph using different colors or styles. For example, if you plot an overlaid graph using different colors for each process factor, you can view the changes in each variable simultaneously on a single graph.

[0141] It is possible to perform multiple plots (Subplots) by arranging multiple graphs as various subgraphs within a single image. This allows for the comparison of multiple data within a single image while maintaining each graph independently.

[0142] After generating each graph individually, multiple graph images can be combined into a single large image using image processing technology (composite graphs). In this case, readability can be optimized through the arrangement and alignment of each graph.

[0143] The effects that can be obtained by converting multiple graphs into a single image are as follows.

[0144] First, by including all relevant data within a single image, the data can be grasped visually at a glance. This can reduce the time users spend analyzing data and drawing conclusions. Combining multiple graphs into a single image makes it easier to simultaneously compare the interrelationships or trends between data sets. This helps in understanding the correlations between the data.

[0145] By integrating multiple graphs into a single image, various process factors and their changes can be grasped at a glance. It allows for the easy discovery of correlations between process factors and the rapid detection of potential issues or anomalies. Representing multiple graphs in a single image simplifies documentation and data sharing, making it particularly useful for report writing and presentations. Combining multiple graphs into a single image using the same style and format maintains visual consistency, providing consistency in data interpretation and reducing user confusion. By leveraging these methods and advantages to integrate multiple graphs into a single image, the efficiency of data analysis and visualization can be significantly enhanced.

[0146] Meanwhile, the processor (150) can generate image data with time as the same axis when multiple graphs are generated in which one of the axes of the graph is time.

[0147] According to one embodiment of the present invention, the processor (150) can detect whether the second product is defective by inputting second image data into an image-based model (S51).

[0148] An image-based model can be trained in various ways; for example, it can be trained using multiple image data indicating a good product and multiple image data indicating a defective product as training data. In this case, image data generated based on at least one process factor affecting whether the product is good or not can be used as training data.

[0149] Meanwhile, an image-based model can be trained to indicate at least one region in the image data that influenced the determination that the product is defective. This is described in detail with reference to Fig. 10.

[0150] According to one embodiment of the present invention, data generated in manufacturing, such as PLCs, is often converted into a numerical form by a data memory map, enabling fast processing and accurate analysis.

[0151] According to one embodiment of the present invention, image data contains information spatially (2D or 3D), so that various data can be reflected on a single image.

[0152] According to one embodiment of the present invention, it can be utilized in various fields such as manufacturing as well as autonomous driving, thus offering excellent versatility.

[0153] According to one embodiment of the present invention, image data usually has high dimensions, so it is suitable for learning complex patterns.

[0154] According to one embodiment of the present invention, in the field of image processing, there are abundant pre-trained models, so high performance can be obtained even with a small amount of data.

[0155] FIG. 9 is a drawing illustrating image data according to an embodiment of the present invention.

[0156] FIG. 9 illustrates image data (400), which is a single image data (400) generated from a plurality of graphs after graphing time series data obtained from each equipment module (equipment module #1, equipment module #2, ..., equipment module #k) from process A to process N.

[0157] FIG. 10 is a drawing illustrating the operation of an electronic device according to one embodiment of the present invention.

[0158] As previously described in relation to S51 in Fig. 8, the image-based model can be trained to indicate at least one area in the image data that influenced the determination that the product is defective.

[0159] A representative method for identifying at least one region that influenced the determination that a product is defective is to use the Class Activation Map (CAM) technique.

[0160] The Class Activation Map technique is a visualization method used in deep learning, particularly in CNNs, that visually indicates which parts of an input image are important when a model makes a prediction. Through this technique, the model's decision-making process can be interpreted. The Class Activation Map generates feature maps for each class using a Global Average Pooling layer before being passed to the final fully connected layer. The feature maps calculate a weighted sum based on class weights and process this using a ReLU function to generate a heatmap that highlights the important areas for the corresponding class. In this case, the heatmap is distinct from the image data itself; in FIG. 10, the important areas (510, 520, 530) that influenced the determination that the image data (500) is defective are depicted in the form of a heatmap. In this case, the higher the weight for an area (the greater the influence), the darker the color is displayed.

[0161] It goes without saying that various techniques, such as Saliency Maps and Occlusion Sensitivity, can be applied in addition to the Class Activation Map technique.

[0162] The processor (150) can identify process information regarding a defect based on at least one second region of the second image data. At this time, the process information may include at least one of the process name where the defect occurred, the equipment module, the time of occurrence, the process factor, and the process value.

[0163] In addition, it may include a product ID or serial number that can uniquely identify a specific product; information on the worker who performed the process (e.g., worker ID, name, shift, etc.); a unique ID of the production line where the defect occurred; equipment status or maintenance records at the time the process was performed; the environment in which the process was performed (e.g., temperature, humidity, cleanliness, etc.); information on the raw materials used in the process, such as the type of raw material, batch number, and supplier; the process step where the defect occurred and its position within the overall process sequence; information on inspection equipment, such as the type and settings of the inspection equipment that detected the defect; setting values ​​of parameters used when performing the process (e.g., temperature settings, pressure settings, etc.); process information and inspection results before and after the process; the date and time when production started and ended; the lot number of the relevant production batch; information on energy consumption (electricity, gas, etc.) used during the process; and production speed and cycle time.

[0164] According to one embodiment of the present invention, by identifying the cause of the defect, it may be easy to devise a countermeasure.

Claims

1. Memory storing at least one instruction; and A processor that executes the above instructions; including The above processor is, Receive time-series data according to the product manufacturing process, and Extract at least one process factor and a plurality of process values ​​corresponding to the at least one process factor from the above time series data, and Image data is generated based on the above at least one process factor and the above plurality of process values, and An electronic device for detecting product defects by inputting the image data into an image-based model to detect whether the product is defective.

2. In Paragraph 1, The above processor is, An electronic device for detecting product defects that generates image data represented as a heatmap based on color values ​​corresponding to the process values.

3. In Paragraph 2, The above processor is, Based on the above image-based model, determine the at least one process factor and the reference color constituting the image data, and An electronic device for detecting product defects by identifying color values ​​corresponding to each process value using the above reference color.

4. In Paragraph 1, The above processor is, An electronic device for detecting product defects that uses normalized multiple process values ​​to generate image data.

5. In Paragraph 3, The above processor is, An electronic device for detecting product defects that identifies color values ​​corresponding to each process value according to a reference color determined based on the data range for each process factor.

6. In Paragraph 1, The above image-based model is, An electronic device for detecting product defects, which is trained to indicate at least one region in image data that influenced the determination that the product is defective.

7. In Paragraph 6, The above processor is, An electronic device for detecting product defects that identifies process information regarding defects based on at least one area of ​​the above image data.

8. In Paragraph 1, The above processor is, Generating at least one graph based on the above at least one process factor and the above plurality of process values, and An electronic device for detecting defects in a product that converts at least one graph into image data.

9. In Paragraph 8, The above processor is, An electronic device for detecting product defects that determines at least one process factor and a variable corresponding to the axis of the graph based on the above image-based model.

10. In Paragraph 9, The above processor is, An electronic device for detecting product defects that converts multiple graphs into a single image data when multiple graphs are generated.

11. A method for detecting product defects performed by an electronic device, A step of receiving time series data according to the product manufacturing process; A step of extracting at least one process factor and a plurality of process values ​​corresponding to the at least one process factor from the time series data; A step of generating image data based on at least one process factor and a plurality of process values; and A method for detecting defects in a product, comprising the step of inputting the image data into an image-based model to detect whether the product is defective.

12. In Paragraph 11, A method for detecting defects in a product that generates image data represented as a heatmap based on color values ​​corresponding to the process values.

13. In Paragraph 12, A step of determining at least one process factor and a reference color constituting the image data based on the above image-based model; and A method for detecting product defects, further comprising the step of identifying a color value corresponding to each process value using the above reference color.

14. In Paragraph 11, A method for detecting defects in a product, further comprising the step of normalizing the plurality of process values ​​and using them to generate the image data.

15. In Paragraph 13, A method for detecting product defects, comprising the step of identifying color values ​​corresponding to each process value according to a reference color determined based on a data range for each process factor.

16. In Paragraph 11, The above image-based model is, A method for detecting product defects, which is trained to indicate at least one region in image data that influences the determination that the product is defective.

17. In Paragraph 16, A method for detecting defects in a product, further comprising the step of identifying process information regarding defects based on at least one region of the image data.

18. In Paragraph 11, A step of generating at least one graph based on the at least one process factor and the plurality of process values; and A method for detecting defects in a product, further comprising the step of converting at least one graph into image data.

19. In Paragraph 18, A method for detecting product defects, further comprising the step of determining at least one process factor and a variable corresponding to the axis of the graph based on the above image-based model.

20. In Paragraph 19, A method for detecting product defects, comprising the additional step of converting multiple graphs into a single image data when multiple graphs are generated.

Images