Electronic device and method for diagnosing performance of motor

Abstract

An electronic device for diagnosing the performance of a motor, according to one embodiment of the present invention, may comprise a processor which: collects time series data and image data according to the starting of a motor; determines whether the motor is abnormal by inputting the time series data to a time series-based model; determines whether the motor is abnormal by inputting the image data to an image-based model; and diagnoses the performance of the motor according to the results of the time series-based model and the image-based model.

Description

Electronic device and method for diagnosing motor performance

[0001] The present invention relates to an electronic device and method for diagnosing the performance of a motor.

[0002] An inverter is a device connected to a motor that controls its operation. If a malfunction occurs in the motor, it can affect the inverter connected to it. If it is determined that the inverter may be damaged due to a motor malfunction, the inverter's operation can be stopped via a trip, thereby halting motor control itself.

[0003] Generally, motors require a high current when first starting and operate at a constant speed after reaching the target speed following an acceleration phase. After operation according to commands, they pass through a deceleration phase and come to a stop. Additionally, in the case of a bidirectional motor, it can operate in both forward and reverse directions.

[0004] Meanwhile, with the commercialization of AI models, there is an increasing number of attempts to utilize them to determine motor abnormalities. Most currently used AI models are single-data models; however, this presents a limitation in terms of reliability, as the results can be easily influenced by environmental factors such as temperature changes and external shocks. Furthermore, since single data only reflects specific types of problems, there is a risk of overlooking other issues.

[0005] Therefore, it is necessary to design an enhanced learning model that can predict the motor's condition from various angles and perform maintenance in advance.

[0006] The objective of the present invention is to provide an electronic device and method that can minimize damage to the motor and quickly detect abnormalities.

[0007] An electronic device for diagnosing the performance of a motor according to an embodiment of the present invention may include a processor that collects time series data and image data according to the start of the motor, inputs the time series data into a time series-based model to determine whether there is an abnormality in the motor, inputs the image data into an image-based model to determine whether there is an abnormality in the motor, and diagnoses the performance of the motor according to the results of the time series-based model and the image-based model.

[0008] The processor can identify a change in the environment or conditions under which the motor operates based on the time series data when the result of the time series-based model determines that the motor is abnormal and the result of the image-based model determines that the motor is normal.

[0009] The processor may determine that a defect exists in any one of the components constituting the motor depending on the type of image data when the motor is determined to be normal as a result of the time-series-based model and the motor is determined to be abnormal as a result of the image-based model.

[0010] The above processor can identify a defective part by using information indicating an associated part for each image data.

[0011] The above processor can provide information on countermeasures based on the fact that the motor has at least one of a component malfunction and an operational malfunction.

[0012] The above processor can determine whether there is an abnormality in the motor using a multimodal model learned based on time series data and image data.

[0013] The above processor can generate the image data using the time series data.

[0014] The above processor can collect image data obtained by photographing the motor.

[0015] A method for diagnosing the performance of a motor performed by an electronic device according to an embodiment of the present invention may include: a step of collecting time series data and image data according to the start of the motor; a step of determining whether there is an abnormality in the motor by inputting the time series data into a time series-based model; a step of determining whether there is an abnormality in the motor by inputting the image data into an image-based model; and a step of diagnosing the performance of the motor according to the results of the time series-based model and the image-based model.

[0016] The step of diagnosing the performance of the motor may include, when the result of the time-series-based model determines that the motor is abnormal and the result of the image-based model determines that the motor is normal, a step of identifying a change in the environment or conditions in which the motor operates based on the time-series data.

[0017] The step of diagnosing the performance of the motor may include, if the result of the time-series-based model determines that the motor is normal and the result of the image-based model determines that the motor is abnormal, determining that a defect exists in any one of the components constituting the motor according to the type of image data.

[0018] The step of diagnosing the performance of the above motor may include a step of identifying a defective part using information indicating an associated part for each image data.

[0019] After the step of diagnosing the performance of the motor, the method may include a step of providing information on countermeasures based on the fact that the motor has at least one of a component defect and an operational defect.

[0020] The above method may further include a step of determining whether there is an abnormality in the motor using a multimodal model learned based on time series data and image data.

[0021] The above-mentioned collecting step may include the step of generating the image data using the above-mentioned time series data.

[0022] The above-mentioned collecting step may include the step of collecting image data obtained by photographing the motor.

[0023] According to one embodiment of the present invention, both the time-series characteristics of the same data and the image characteristics through preprocessing can be utilized.

[0024] According to one embodiment of the present invention, motor predictive maintenance can be divided into multiple tracks through the evaluation of time-series / image characteristics of the same data.

[0025] According to one embodiment of the present invention, since a time-series based model is easy to identify data flow and an image-based model is easy to identify data features, an abnormal situation can be estimated when the results of the time-series and image-based models are different.

[0026] FIG. 1 is a schematic diagram illustrating a motor performance diagnostic system according to one embodiment of the present invention.

[0027] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0028] FIG. 3 is a diagram illustrating the operation flowchart of an electronic device according to one embodiment of the present invention.

[0029] FIG. 4 is a diagram illustrating the results of applying a model according to one embodiment of the present invention.

[0030] FIG. 5 is a drawing illustrating image data according to the first embodiment of the present invention.

[0031] FIG. 6 is a drawing illustrating image data according to a second embodiment of the present invention.

[0032] FIG. 7 is a drawing illustrating the application of a model according to the first embodiment of the present invention.

[0033] FIG. 8 is a drawing illustrating the application of a model according to a second embodiment of the present invention.

[0034] FIG. 9 is a drawing illustrating the operation of an electronic device according to one embodiment of the present invention.

[0035] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiment in which the present invention can be practiced. In order to clearly explain the present invention in the drawings, parts unrelated to the description may be omitted, and the same reference numerals may be used for identical or similar components throughout the specification.

[0036] FIG. 1 is a schematic diagram illustrating a motor performance diagnostic system according to one embodiment of the present invention.

[0037] The motor performance diagnostic system (1) of FIG. 1 may include an inverter (10), a motor (20), and an electronic device (100).

[0038] An inverter (10) is a device connected to a motor (20) to control the operation of the motor (20). The inverter (10) and the motor (20) can be connected to each other through various methods such as single-phase connection, three-phase connection, and pulse width modulation (PWM), and the inverter (10) may include a configuration for collecting control data according to the start of the motor (20). At this time, the control data may include current data, frequency data, etc. of the motor (20) as time-series data.

[0039] The electronic device (100) is a device that diagnoses the performance of the motor (20) using control data and can be implemented as a computer, PLC (Programmable Logic Controller), server, smartphone, tablet PC, smart pad, laptop, etc. In addition, unlike what is shown in FIG. 1, the electronic device (100) can be implemented as a component of the inverter (10), and the method of implementation of the electronic device (100) is not limited to any one. However, for the convenience of the following explanation, the inverter (10) and the electronic device (100) are assumed to be separate devices.

[0040] In the present invention, a method for diagnosing the performance of a motor (20) is proposed by utilizing two or more models trained to diagnose the performance of the motor (20) based on image data that has been converted into an image of the control data, as well as control data.

[0041] Hereinafter, the configuration and operation of an electronic device (100) according to one embodiment of the present invention will be described in detail with reference to the drawings.

[0042] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0043] An electronic device (100) according to one embodiment of the present invention may include an input unit (110), a communication unit (120), a display unit (130), a storage unit (140), and a processor (150).

[0044] The input unit (110) generates input data in response to user input of the electronic device (100). For example, the user input may be a user input requesting the operation of a program for diagnosing motor performance, a user input for preprocessing control data, etc. In addition to this, any user input necessary to identify whether the motor (20) is abnormal and to diagnose the performance of the motor (20) can be applied without limitation.

[0045] The input unit (110) includes at least one input means. The input unit (110) may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a mouse, a menu button, etc.

[0046] The communication unit (120) can communicate with external devices such as a server, an inverter (10) to transmit and receive control data, information about the starting section, frequency change rate, multiple learning models, etc.

[0047] To this end, the communication unit (120) can perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Wi-Fi (wireless fidelity), Bluetooth, or wired communication such as LAN (local area network), WAN (Wide Area Network), and power line communication.

[0048] The display unit (130) displays display data according to the operation of the electronic device (100). The display unit (130) may display a screen showing frequency changes over time, a screen showing the starting section, a screen showing whether there is an abnormality in the motor and its performance, and other screens receiving user input.

[0049] The display unit (130) includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (130) can be combined with the input unit (110) to be implemented as a touch screen.

[0050] The storage unit (140) stores operation programs of the electronic device (100). The storage unit (140) includes storage with non-volatile properties that can preserve data (information) regardless of whether power is provided, and memory with volatile properties in which data to be processed by the processor (150) is loaded and data cannot be preserved if power is not provided. Storage includes flash memory, hard-disc drive (HDD), solid-state drive (SSD), and ROM (Read Only Memory), and memory includes buffer and RAM (Random Access Memory).

[0051] The storage unit (140) can store control data, image data, multiple learning models, results of the first model and the second model, etc. The storage unit (140) can store computation programs, etc., necessary during the process of collecting control data, converting image data, learning models, identifying abnormalities in the motor, and diagnosing the performance of the motor.

[0052] The processor (150) can control at least one other component (e.g., hardware or software component) of the electronic device (100) by executing software such as a program, and can perform various data processing or operations.

[0053] A processor (150) according to one embodiment of the present invention collects time series data according to the start of the motor, converts the time series data into image data, inputs the time series data into a time series-based model to determine whether there is an abnormality in the motor, inputs the image data into an image-based model to determine whether there is an abnormality in the motor, and can diagnose the performance of the motor according to the time series-based model and the image-based model.

[0054] At this time, the processor (150) may train a plurality of models to determine whether there is an abnormality in the motor, or receive and store a plurality of previously trained and built models from the outside and use them, and is not limited to either one.

[0055] Meanwhile, the processor (150) may perform at least some of the data analysis, processing, and result information generation for performing the above operations using at least one of machine learning, neural network, or deep learning algorithms as a rule-based or artificial intelligence algorithm. Examples of neural networks may include models such as CNN (Convolutional Neural Network), DNN (Deep Neural Network), and RNN (Recurrent Neural Network).

[0056] FIG. 3 is a diagram illustrating the operation flowchart of an electronic device according to one embodiment of the present invention.

[0057] According to one embodiment of the present invention, the processor (150) can collect time series data and image data according to the operation of the motor (20) (S10).

[0058] Time series data is data received from sensors attached to the inverter (10) and the motor (20), and is data acquired over time. The sensors may include, for example, a current sensor, a voltage sensor, a temperature sensor, a vibration sensor, a rotational speed / position sensor, etc. The current sensor measures the current of the motor (20) in real time, and when the motor (20) is connected to the inverter (10) in three phases, the current data may include current data for each phase (e.g., U, V, W phases).

[0059] At this time, the processor (150) may collect time-series data from the inverter (10), but the collection path is not limited to any one, such as through a server. In addition, the type and number of the collected time-series data are not limited to any one.

[0060] In the present invention, a method is proposed to examine the abnormality of the motor (20) from various angles by utilizing a model based on image data as well as time series data.

[0061] Image data according to one embodiment of the present invention may include data generated using time series data and data obtained by photographing the exterior of the motor (20).

[0062] First, the processor (150) can generate image data using time series data. Various methods can be applied to convert time series data into image data and are not limited to any one. For example, each data value of the time series data can be converted into an image as a color heatmap so that it is mapped to a unique color, or the data values ​​obtained by preprocessing the time series data can be graphed.

[0063] Additionally, the processor (150) can examine whether there is an abnormality in the motor (20) by using image data that captures the exterior of the motor (20) itself. For example, image data can be obtained through a vision camera, an infrared camera, etc. placed on a process line. The means of obtaining image data is not limited to any one. It may be possible to check whether a part is properly mounted through a vision camera, or to check the temperature through an infrared camera.

[0064] In addition, image data can correspond one-to-one with time-series data, but is not limited thereto and can correspond many-to-one. For example, current data (time-series data) can be converted into image data. As another example, current data and voltage data (time-series data) can be converted into a single image data. An example of image data is illustrated in FIGS. 5 and FIGS. 6.

[0065] According to one embodiment of the present invention, the processor (150) can determine whether there is an abnormality in the motor (20) by inputting time series data into a time series-based model (S20).

[0066] The time series-based model is an artificial intelligence model trained to determine whether the motor (20) is abnormal based on time series data. It may include, but is not limited to, various types of artificial intelligence models such as CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based Deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), LSTM (Long Short-Term Memory), transformer, and Gated Recurrent Unit (GRU). In addition, regression and clustering may be applied.

[0067] A time series-based model can be created for each type of time series data and can be adopted and used as one depending on the type of time series data collected. Additionally, a time series-based model can be trained based on two or more types of time series data. For example, a time series-based model can be trained to determine whether there is an abnormality in the motor (20) using current data. As another example, a time series-based model can be trained to determine whether there is an abnormality in the motor (20) using current data and voltage data.

[0068] According to one embodiment of the present invention, the processor (150) can determine whether there is an abnormality in the motor (20) by inputting image data into an image-based model (S30).

[0069] The image-based model is an artificial intelligence model trained to determine whether there is an abnormality in the motor (20) based on image data, and may include, but is not limited to, various types of artificial intelligence models such as CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), R-CNN (Region with Convolutional Neural Network), S-DNN (Stacking-based deep Neural Network), RPN (Region Proposal Network), DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory), and Transformer.

[0070] Image-based models can be created for each type of image data and can be adopted and used as one depending on the type of image data collected. In addition, image-based models can be trained based on two or more types of image data.

[0071] For example, an image-based model can be trained to determine whether there is an abnormality in the motor (20) using image data obtained by converting current data (time series data) into an image. As another example, an image-based model can be trained to determine whether there is an abnormality in the motor (20) using image data obtained by converting current data and voltage data (time series data) into an image. In this case, the image data may be obtained by converting the current data and voltage data (time series data) into image data separately, or it may be obtained by combining the time series data and converting them into a single image data.

[0072] As another example, an image-based model may be trained to determine whether there is an abnormality in the appearance of a specific area of ​​the motor (20) using image data taken of that area of ​​the motor (20).

[0073] According to one embodiment of the present invention, the processor (150) can diagnose the performance of the motor (20) based on the results of the time series-based model and the image-based model (S40).

[0074] If the processor (150) determines that both the time-series based model and the image-based model are normal, the performance of the motor (20) can be determined to be normal.

[0075] If the processor (150) determines that both the time series-based model and the image-based model are abnormal, it can identify the cause through the time series data and the image data, respectively.

[0076] Among these, the cases where only the results of the time-series-based model are abnormal and the cases where only the results of the image-based model are abnormal are explained with reference to Fig. 4.

[0077] According to one embodiment of the present invention, both the time-series characteristics of the same data and the image characteristics through preprocessing can be utilized.

[0078] According to one embodiment of the present invention, motor predictive maintenance can be divided into multiple tracks through the evaluation of time-series / image characteristics of the same data.

[0079] FIG. 4 is a diagram illustrating the results of applying a model according to one embodiment of the present invention.

[0080] As previously described with reference to S40 in FIG. 3, the performance of the motor (20) according to the results of the time series-based model and the image-based model is shown in the table.

[0081] First, the processor (150) may determine that an operational abnormality has occurred in the motor (20), such as when the result motor (20) of the time-series-based model is determined to be abnormal (F) and the result motor (20) of the image-based model is determined to be normal (T).

[0082] The time series-based model is a model that learns the characteristics of the motor (20) through numerical changes over time and is more specialized in detecting operational abnormalities.

[0083] For example, when the operating conditions of the motor (20) change, such as when the motor (20) starts in the forward direction and then starts in the reverse direction, the load of the motor may increase rapidly or the speed may change rapidly. When the load of the motor increases or the speed accelerates / decels rapidly, the value of the current data increases. In this case, the processor (150) can detect that an abnormality has occurred through the change in the value of the current data obtained from the current sensor of the motor (20).

[0084] As another example, when the motor operating environment changes, such as the intrusion of foreign matter (increasing the load of the motor (20)), an abnormality can be detected through the time-series numerical change of the current data. As yet another example, when the mechanical load of the motor (20) increases as the temperature rises, an abnormality can be detected through the time-series numerical change of the current data at a specific point in time.

[0085] The processor (150) can identify changes in the environment or conditions in which the motor (20) is operated based on time series data. For example, by using time series data obtained from the vibration sensor of the motor (20), it is possible to determine whether vibration is occurring at the installation location of the motor (20).

[0086] The processor (150) can identify time series data with high correlation by performing a correlation analysis between the input time series data and the model judgment result. Correlation analysis is a statistical method for determining the strength and direction of the relationship between two variables. The results of the correlation analysis can be expressed as how closely the two variables are related to each other (strength), whether the two variables move in the same direction (positive correlation), or whether they move in opposite directions (negative correlation).

[0087] Accordingly, the processor (15) can identify the type of time series data that has a high correlation (e.g., ±0.7 or higher) with the result determined to be an abnormality of the motor (20), and identify the cause of the abnormality in the operation of the motor (20).

[0088] Meanwhile, if the processor (150) determines that the motor (20) resulting from the time-series-based model is normal (T) and the motor (20) resulting from the image-based model is abnormal (F), it can determine that a defect exists in any one of the components constituting the motor (20) depending on the type of image data.

[0089] The image-based model is a model that learns the characteristics of the motor (20) through changes in specific areas of the image and is more specialized in detecting abnormalities on specific parts. Depending on the sensor data and preprocessing method, the specific types of abnormalities may vary.

[0090] For example, the DQ-Lissajous waveform image of a current sensor is useful for identifying abnormalities in the motor input current phase imbalance in a rotating coordinate system (DQ coordinate system). The DQ coordinate system represents the result of converting variables such as three-phase current and voltage in the motor (20) into two rectified axes (d-axis: DC axis, q-axis: Cartesian axis). The Lissajous waveform is used to visualize the relationship between the d-axis and q-axis variables in this DQ coordinate system. Changes in the DQ-Lissajous waveform image can be seen as an abnormality occurring in the stator or rotor of the motor (20).

[0091] Changes in the image obtained by performing a Fast Fourier Transform (FFT) on the time-series data of a vibration sensor are useful for identifying bearing defects. The Fourier-transformed data can be graphed in the Power Spectrum Density (PSD) format to generate image data.

[0092] The processor (150) can identify a defective part by using information indicating an associated part for each image data. The information indicating an associated part for each image data may include the type of image data and the type of part for which an abnormality can be identified from the image data.

[0093] For example, the DQ-Lissajous waveform image data of the current sensor is related to whether there are abnormalities in the stator and rotor of the motor (20). The DQ-Lissajous waveform is a visual representation of the relationship between the d-axis and q-axis current components in the DQ coordinate system. The d-axis and q-axis current components can be obtained by converting the three-phase current of the electromechanical system into the DQ coordinate system. The Lissajous waveform is a graph representing the phase relationship and amplitude ratio between two signals, which allows for the visual identification of abnormal current phenomena.

[0094] Abnormal conditions in the stator and rotor primarily manifest through current imbalances or phase shifts. DQ-Lissajous waveforms can intuitively display these current changes and imbalances. Damage or abnormalities in the stator or rotor appear as changes in the shape of the DQ-Lissajous waveform. For example, a normal signal forms a consistent ellipse or circle, but if an abnormality occurs, this shape becomes distorted or changes into an irregular form. Therefore, the motor's condition can be monitored in real time via DQ-Lissajous waveforms, enabling fast and effective maintenance.

[0095] As another example, the Fourier transform image of a vibration sensor is related to the detection of bearing abnormalities. The Fourier transform is a method that converts a signal into the frequency domain to analyze the magnitude and phase of each frequency component. By applying the Fourier transform to vibration or noise signals measured by a vibration sensor, an image visualizing the frequency components can be generated.

[0096] The presence of bearing abnormalities can primarily be identified through the analysis of the frequency components of vibrations or noise. Abnormal frequency components appear when a bearing is damaged or worn. Fourier transform images visually display the frequency components of a vibration signal. While normal bearings exhibit a specific frequency pattern, this pattern changes or new frequency components appear when an abnormality occurs. Through the Fourier transform, the frequency range of abnormal vibrations or noise can be accurately identified. For example, strong vibrations at a specific frequency may indicate a particular bearing defect.

[0097] In summary, abnormalities in the stator and rotor can be visually and quickly and accurately identified through DQ-Lissajous waveforms regarding current phase imbalances and changes, while bearing abnormalities can be diagnosed in detail by analyzing the frequency components of vibration or noise through Fourier transform images.

[0098] Additionally, the processor (150) can identify image data with high correlation by performing a correlation analysis between the input image data and the model judgment result. A Class Activation Map (CAM) technique is used on the image data, and the heatmap produced through the CAM technique can be normalized to between 0 and 1.

[0099] Accordingly, the processor (15) can identify the type of image data that has a high correlation with the result determined to be abnormal in the motor (20) (e.g., a normalized value of ±0.7 or higher) and identify the defective part among the parts constituting the motor (20). Specific details regarding this are described with reference to FIG. 9.

[0100] According to one embodiment of the present invention, since a time-series based model is easy to identify data flow and an image-based model is easy to identify data features, an abnormal situation can be estimated when the results of the time-series and image-based models are different.

[0101] According to one embodiment of the present invention, depending on the abnormal situation of the motor (20), different countermeasures may be provided to the user. The processor (150) may provide information on countermeasures based on the motor (20) having at least one of a component abnormality and an operational abnormality. Information on countermeasures may be provided, for example, through a display unit (130), or may be displayed on a worker terminal managing the motor (20).

[0102] In the event of an operational anomaly, response measures based on the aspects and patterns of the data can be provided, for example, "Is the motor not running?", "It seems an abnormal object is stuck in the motor", or "Has the operating method changed?"

[0103] In the event of a component malfunction, countermeasures can be provided, for example, "Noise is occurring in a specific band of vibration. It appears to be a motor bearing defect. Please check the bearing," or "Noise is occurring in a specific band of current. Please check if the motor coil has been exposed to overcurrent."

[0104] FIG. 5 is a drawing illustrating image data according to the first embodiment of the present invention.

[0105] The image data (500) of Fig. 5 is an image created by converting each data value of the time series data into a color heatmap so that it is mapped to a unique color.

[0106] The processor (150) can generate image data (500) by scaling the data values ​​of the time series data within a minimum-maximum range and mapping corresponding colors within each unit area. At this time, the types and number of data included in the image data (500) are not limited to any one.

[0107] FIG. 6 is a drawing illustrating image data according to a second embodiment of the present invention.

[0108] The image data (600) of Fig. 6 is a graph of data values ​​obtained by preprocessing time series data.

[0109] FIG. 7 is a drawing illustrating the application of a model according to the first embodiment of the present invention.

[0110] As described above with reference to FIG. 3, time series data (710) can be input into a time series-based model (720) to obtain a first prediction result (730).

[0111] Additionally, image data (740) can be input into an image-based model (750) to obtain a second prediction result (760). The processor (150) can diagnose the performance of the motor (20) by comparing the first prediction result (730) and the second prediction result (760).

[0112] FIG. 8 is a drawing illustrating the application of a model according to a second embodiment of the present invention.

[0113] FIG. 8 illustrates a multimodal model (830) that utilizes both time series data (810) and image data (820).

[0114] The processor (150) can determine whether the motor (20) is abnormal by using a multimodal model learned based on time series data and image data.

[0115] Multimodal models are models trained using different types of time-series data and image data, and can be classified into Early Fusion, Late Fusion, and Hybrid Fusion depending on the data combination method.

[0116] Early fusion is a method that combines time-series data and image data during the data input stage to process them into a single integrated input. It has the advantage that the model can learn the interactions between the data early on.

[0117] Post-fusion is a method that processes time-series data and image data independently and then combines the results to generate a final output. It has the advantage of enabling network designs specialized for each modal.

[0118] Hybrid fusion combines early and late fusion, where some features are combined at the input stage while others are processed independently before being combined. It offers the advantages of flexibility and the ability to better capture interactions between data.

[0119] FIG. 9 is a drawing illustrating the operation of an electronic device according to one embodiment of the present invention.

[0120] As described above with reference to FIG. 4, if the result of the image-based model determines that the motor (20) is abnormal (F), it can be determined that a defect exists in any one of the components constituting the motor (20) depending on the type of image data.

[0121] At this time, the image-based model can be trained to mark at least one region in the image data that influenced the determination that the product is defective. As a representative method for marking at least one region that influenced the determination that the product is defective, the Class Activation Map (CAM) technique can be used.

[0122] The Class Activation Map technique is a visualization method used in deep learning, particularly in CNNs, that visually indicates which parts of an input image are important when a model makes a prediction. Through this technique, the model's decision-making process can be interpreted. The Class Activation Map generates feature maps for each class using a Global Average Pooling layer before being passed to the final fully connected layer. The feature maps calculate a weighted sum based on class weights and process this using a ReLU function to generate a heatmap that highlights the areas important to the corresponding class. In this case, the heatmap is distinct from the image data itself; in FIG. 9, important areas (911, 912, 913) that influenced the determination that the image data (901) is defective, and important areas (921, 922, 923) that influenced the determination that the image data (902) is defective are depicted in the form of a heatmap. In this case, the higher the weight for an area (the greater the influence), the darker the color is displayed.

[0123] It goes without saying that various techniques, such as Saliency Maps and Occlusion Sensitivity, can be applied in addition to the Class Activation Map technique.

[0124] According to one embodiment of the present invention, by identifying the cause of the defect, it may be easy to devise a countermeasure.

Claims

1. In an electronic device for diagnosing the performance of a motor, Collecting time-series data and image data based on the start of the above motor, The above time series data is input into a time series-based model to determine whether there is an abnormality in the motor, and The above image data is input into an image-based model to determine whether there is an abnormality in the motor, and An electronic device comprising a processor that diagnoses the performance of a motor according to the results of the above time series-based model and the above image-based model.

2. In Paragraph 1, The above processor is, An electronic device that identifies a change in the environment or conditions in which the motor operates based on the time series data when the result of the time series-based model determines that the motor is abnormal and the result of the image-based model determines that the motor is normal.

3. In Paragraph 1 or 2, The above processor is, An electronic device that determines that a defect exists in any one of the components constituting the motor according to the type of image data when the result of the above time-series-based model determines that the motor is normal and the result of the above image-based model determines that the motor is abnormal.

4. In Paragraph 3, The above processor is, An electronic device that identifies a defective part using information indicating an associated part for each image data.

5. In Paragraph 1, The above processor is, An electronic device that provides information on countermeasures based on the fact that the above motor has at least one of a component defect and an operational defect.

6. In Paragraph 1, The above processor is, An electronic device that determines whether a motor is abnormal using a multimodal model learned based on time series data and image data.

7. In Paragraph 1, The above processor is, An electronic device that generates the image data using the above time series data.

8. In Paragraph 1, The above processor is, An electronic device that collects image data obtained by photographing the above motor.

9. A method for diagnosing the performance of a motor performed by an electronic device, A step of collecting time-series data and image data according to the start of the above motor; A step of determining whether there is an abnormality in the motor by inputting the above time series data into a time series-based model; A step of determining whether there is an abnormality in the motor by inputting the above image data into an image-based model; A method comprising the step of diagnosing motor performance based on the results of the above time series-based model and the above image-based model.

10. In Paragraph 9, The step of diagnosing the performance of the above motor is, A method comprising the step of identifying a change in the environment or condition in which the motor operates based on the time series data, when the result of the time series-based model determines that the motor is abnormal and the result of the image-based model determines that the motor is normal.

11. In Paragraph 9 or 10, The step of diagnosing the performance of the above motor is, A method comprising the step of determining that a defect exists in any one of the components constituting the motor according to the type of image data, when the result of the time-series-based model determines that the motor is normal and the result of the image-based model determines that the motor is abnormal.

12. In Paragraph 11, The step of diagnosing the performance of the above motor is, A method comprising the step of identifying a defective part using information indicating an associated part for each image data.

13. In Paragraph 9, After the step of diagnosing the performance of the above motor, A method comprising the step of providing information on countermeasures based on the fact that the motor has at least one of a component defect and an operational defect.

14. In Paragraph 9, A method further comprising the step of determining whether there is an abnormality in the motor using a multimodal model learned based on time series data and image data.

15. In Paragraph 9, The above-mentioned collecting step is, A method comprising the step of generating image data using the above time series data.

16. In Paragraph 9, The above-mentioned collecting step is, A method comprising the step of collecting image data obtained by photographing the motor.

Images