Electronic device for controlling operation on basis of sensor value, method therefor, and computer-readable storage medium

Abstract

This electronic device may comprise: a memory storing instructions and including one or more storage media; and at least one processor including processing circuitry. The electronic device may receive sensing data from a plurality of sensors, and compare and analyze, in real time, the similarity with sensing data of a second sensor among the plurality of sensors to determine whether a first sensor has a failure. The electronic device may determine a failure type of the first sensor on the basis of the failure of the first sensor being determined. The electronic device may perform at least one of masking, weight adjustment, or interpolation on sensing data of the first sensor on the basis of the determined failure type to generate input data of a multi-sensor-based prediction model, and control inference of the prediction model to be performed by using the generated input data.

Description

Electronic device for controlling operation based on sensor values, method thereof, and computer-readable storage medium

[0001] This document relates to an electronic device that controls operation based on sensor values, a method thereof, and a storage medium. For example, it relates to an electronic device, a storage medium, and a method thereof that controls the heating operation of a heating element based on sensor values.

[0002] Multi-sensor-based multi-modality models can have a problem where the occurrence of defect data affects the prediction of the final result, thereby degrading the accuracy of the final model prediction. In such cases, defects were handled by ensuring safety through the deletion of defect data or the suspension of the prediction model.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] While ensuring safety by deleting fault data or halting predictive models is effective in the short term, it carries drawbacks such as potential data loss or model performance degradation, and the inability to provide stable service support, which could lead to service interruptions. Therefore, it may be necessary to manage fault data in real-time through fault detection and utilize alternative data when required.

[0005] The electronic device and method for controlling operation based on sensor values ​​according to the present document disclose a real-time fault detection monitoring method, and can improve the accuracy of the model by selectively applying a method to improve accuracy according to the type of fault.

[0006] The electronic device and method for controlling operation based on sensor values ​​according to this document can compensate for defects through other sensors or predictive models even if a sensor defect occurs through real-time fault detection and the use of alternative data. This allows the stability of the electronic device to be maintained and data to be provided consistently, enabling continuous operation without interruption.

[0007] The electronic device may include a memory that stores instructions and includes one or more storage media, and at least one processor that includes processing circuitry. The electronic device may receive sensing data from a plurality of sensors and determine whether a first sensor is faulty by comparing and analyzing the similarity with the sensing data of a second sensor among the plurality of sensors in real time. Based on the confirmation that a fault has been found in the first sensor, the electronic device may determine the type of failure of the first sensor. Based on the determined type of failure, the electronic device may perform at least one of masking, weighting, or interpolation on the sensing data of the first sensor to generate input data for a multi-sensor-based prediction model, and control the prediction model to perform inference using the generated input data.

[0008] A computer-readable non-transient storage medium storing one or more programs including instructions executable by a processor can receive sensing data from a plurality of sensors, compare and analyze the similarity with the sensing data of a second sensor among the plurality of sensors in real time to determine whether a first sensor is faulty, and control an electronic device to determine the type of fault of the first sensor based on the confirmation that the first sensor is faulty. The computer-readable non-transient storage medium can control an electronic device to generate input data for a multi-sensor-based prediction model by performing at least one of masking, weighting, or interpolation on the sensing data of the first sensor based on the determined type of fault, and to perform inference of the prediction model using the generated input data.

[0009] A method of operation of an electronic device may include receiving sensing data from a plurality of sensors, comparing and analyzing the similarity with the sensing data of a second sensor among the plurality of sensors in real time to determine whether the first sensor is faulty, and determining the type of fault of the first sensor based on the confirmation that the first sensor is faulty. A method of operation of the electronic device may include generating input data for a multi-sensor-based prediction model by performing at least one of masking, weighting, or interpolation on the sensing data of the first sensor based on the determined type of fault, and performing inference of the prediction model using the generated input data.

[0010] The electronic device and method for controlling operation based on sensor values ​​according to this document can improve the phenomenon of reduced accuracy that may occur when a failure of a specific sensor in a multi-modality model using various sensors is detected.

[0011] The electronic device and method for controlling operation based on sensor values ​​according to this document can improve accuracy by adding logic to adjust weights within the model when an anomaly is detected.

[0012] The electronic device and method for controlling operation based on sensor values ​​according to this document can improve sensor data quality and enhance the accuracy and reliability of the model.

[0013] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0014] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0015] Figure 2 illustrates data measured in a situation where a malfunction occurred in the sensor.

[0016] Figure 3 is a graph showing the azimuth information of the GPS (global positioning system) sensor and the gyro sensor according to speed information.

[0017] FIG. 4 illustrates the process of an electronic device clipping sensor data according to one embodiment.

[0018] FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0019] FIG. 6 is a flowchart illustrating the operation of an electronic device according to one embodiment checking for abnormalities in acceleration data using a GPS sensor.

[0020] FIG. 7 is a flowchart illustrating the process of an electronic device according to one embodiment determining whether a gyro sensor is faulty.

[0021] FIG. 8 illustrates the process of reducing the influence of defect data by adding a mask layer to an electronic device according to one embodiment.

[0022] FIG. 9 illustrates the process of an electronic device according to one embodiment improving the accuracy of a model by failure type.

[0023] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0024] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0025] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0026] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0027] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0031] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0032] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0033] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0037] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0039] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0042] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0044] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] According to one embodiment, the number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core. The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0047] Figure 2 illustrates data measured in a situation where a malfunction occurred in the sensor.

[0048] According to one embodiment, the electronic device (101) can verify data measured in a situation where a complete failure of the sensor has occurred, as shown in FIG. 2. FIG. 2 is a graph showing the measured values ​​of the acceleration sensor over time, where the X-axis represents time (seconds) and the Y-axis represents acceleration (m / s²). 2 It can mean ).

[0049] According to one embodiment, the electronic device (101) may detect a phenomenon in which sensor data exhibits a dynamically changing pattern during normal operation, and then the sensor value becomes fixed at a constant value during a stuck section (210) where a stuck phenomenon occurs. The stuck phenomenon may refer to a failure that may occur in an inertial sensor of the MEMS (micro electro mechanical systems) structure of the electronic device (101). The stuck phenomenon may refer to a state in which the sensor is fixed at a specific position and does not move due to external shock, excessive acceleration, or static electricity.

[0050] According to one embodiment, the electronic device (101) may utilize a statistical method to detect sensor failure in real time. For example, the electronic device (101) may check the variability of the data by calculating the standard deviation or variance of the sensor data over a continuous time interval. The electronic device (101) may be able to confirm that the standard deviation is maintained above a certain level during normal operation, but when a stuck phenomenon occurs, the standard deviation decreases rapidly and approaches almost zero.

[0051] According to one embodiment, the electronic device (101) can identify outliers in the sensor data. An outlier may refer to a value that deviates from a specified level in the pattern of the entire data. The electronic device (101) can identify outliers in the sensor data using the Z-score and interquartile range (IQR) methods.

[0052] According to one embodiment, a jet-score may refer to a method for measuring how far data is from the mean. An electronic device (101) may determine that data is an outlier if it deviates from the mean by a certain range. The electronic device (101) may use the jet-score when the distribution of data follows a normal distribution.

[0053] According to one embodiment, the interquartile range method may refer to a method of detecting outliers by dividing the whole into four equal parts when the data is arranged in order of size. The electronic device (101) may determine that a data value is an outlier if it falls outside the range set based on the upper 25% point and the lower 25% point of the overall distribution. The electronic device (101) can reliably detect outliers regardless of the distribution shape of the data by using the interquartile range method. The electronic device (101) can continuously check for anomalies in sensor data by applying these statistical methods in real time. The electronic device (101) can adjust the sensitivity of outlier detection by setting an appropriate reference value according to the characteristics of the sensor and the usage environment.

[0054] According to one embodiment, when a stuck phenomenon such as that shown in FIG. 2 occurs, the electronic device (101) can classify the corresponding failure as a first type (e.g., complete failure) and execute an appropriate response. For example, the electronic device (101) can process the corresponding sensor data by completely masking it or by switching it to a pre-trained preliminary model.

[0055] According to one embodiment, the electronic device (101) may perform monitoring for a set time (e.g., 1 hour) or attempt to recover the sensor through a power reset to determine whether the stuck phenomenon is temporary or permanent. The set time is merely an example and is not a fixed value. If the sensor does not return to a normal operating range within the set time, or if the power reset attempt exceeds a specific number of times (e.g., 10 times), the electronic device (101) may determine the sensor to be in a completely failed state and switch to a replacement sensor or a spare model. The specific number is merely an example and may vary depending on the setting.

[0056] Figure 3 is a graph showing the azimuth information of the GPS sensor and the gyro sensor according to speed information.

[0057] According to one embodiment, the electronic device (101) can compare and analyze the speed information of an object with the azimuth information measured from the GPS sensor and the gyroscope sensor as shown in FIG. 3. The upper graph (310) of FIG. 3 may show the change in the speed of the object over time. The lower graph (320) may display the azimuth information measured from the GPS sensor and the gyroscope sensor at the same time. The electronic device (101) can confirm that from the point (312) (e.g., about 752 seconds) when the speed of the object increases to 10 km / h or more, the azimuth data from the GPS sensor begins to show a pattern similar to the measurement value from the gyroscope sensor.

[0058] According to one embodiment, the electronic device (101) can identify conditions under which fault detection of the gyroscope sensor using the GPS sensor can be reliably performed through this analysis. The electronic device (101) can determine that fault detection of the gyroscope sensor using the GPS sensor is reliable when the object is traveling at a specified speed (e.g., 10 km / h) or higher. Based on the condition of the specified speed being satisfied, the electronic device (101) can determine whether the gyroscope sensor is faulty by comparing the change in azimuth of the GPS sensor with the integral value of the angular velocity of the gyroscope sensor.

[0059] According to FIG. 3, it can be seen that the azimuth data of the two sensors exhibit a similar pattern even in the section where the direction of movement of the object changes rapidly (e.g., about 752 seconds). The electronic device (101) can use the azimuth data of the GPS sensor as backup data for the gyroscope sensor during high-speed driving. According to one embodiment, the electronic device (101) can evaluate the data reliability of the gyroscope sensor based on the azimuth data of the GPS sensor in the section where the accuracy of the GPS signal exceeds a specified level, the object travels in a straight line, and moves at a constant speed.

[0060] FIG. 4 illustrates the process of an electronic device clipping sensor data according to one embodiment.

[0061] According to one embodiment, the electronic device (101) may perform clipping processing as shown in FIG. 4 when a second type of failure occurs in which the sensor data exceeds a set operating range. The graph in FIG. 4 may include the original value and the clipped value of the sensor data over time. The upper and lower threshold values ​​(+0.5 and -0.5, respectively) indicated by dotted lines may represent the acceptable operating range of the sensor data. The magnitudes of the upper and lower threshold values ​​are not fixed and may vary depending on the settings.

[0062] According to one embodiment, the electronic device (101) may perform clipping processing to limit the original sensor data to the nearest threshold when the original sensor data exceeds an upper threshold (e.g., +0.5) or becomes smaller than a lower threshold (e.g., -0.5). The size of the threshold is merely an example and may vary depending on the settings. For example, the electronic device (101) may clip the sensor data to the lower threshold of -0.5 when the sensor data records a value smaller than the lower threshold (-0.5) (near -1.0) around 4 seconds on the time axis. The electronic device (101) may also limit the sensor data to the upper threshold of +0.5 when the sensor data exceeds the upper threshold (+0.5) around 7 seconds.

[0063] According to one embodiment, the electronic device (101) can remove extreme values ​​of sensor data through clipping processing while maintaining the overall pattern and trend of the data. The electronic device (101) can limit the range so that the clipped data follows the change pattern of the original data but changes only within a set operating range. The electronic device (101) can minimize the impact of temporary abnormal values ​​of the sensor on the performance of the entire system through clipping processing.

[0064] According to one embodiment, the electronic device (101) can dynamically set a threshold value for clipping processing based on the characteristics of the sensor and the application field. For example, in the case of an accelerometer, the electronic device (101) can set a threshold value by considering the maximum acceleration value expected in a normal usage environment. In the case of a gyroscope, the electronic device (101) can determine a threshold value based on the expected maximum angular velocity range. The electronic device (101) can dynamically adjust the threshold value according to the operating mode of the system or the activity state of the user.

[0065] According to one embodiment, the electronic device (101) can evaluate the overall condition of the sensor by monitoring the rate of clipped data. If the frequency of clipping occurring within a specific time interval exceeds a set threshold value, the electronic device (101) may determine that this indicates a serious malfunction of the sensor. In this case, the electronic device (101) may completely mask the data of the sensor or switch to an alternative sensor. Through this, the electronic device (101) can ensure stable operation of the system while continuously maintaining the reliability of the sensor data.

[0066] FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0067] The operations described through FIG. 5 may be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (500) may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 4, and the technical features described above will be omitted below. The order of each operation in FIG. 5 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0068] According to one embodiment, operations 510 to 540 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0069] According to one embodiment, in operation 510, the electronic device (101) can receive and process sensing data from a plurality of sensors. The plurality of sensors may include at least one of an accelerometer, a gyroscope, a GPS (global positioning system) sensor, a microphone, or a camera. The types of sensors are merely examples and are not limited thereto. The electronic device (101) can receive and process data in real time from each sensor through a sensor driver.

[0070] According to one embodiment, in operation 520, the electronic device (101) can determine whether the first sensor is faulty by comparing and analyzing the similarity between the sensing data of the first sensor and the sensing data of the second sensor in real time. For example, the electronic device (101) can compare the acceleration calculated from the location data of the GPS sensor with the measurement value of the acceleration sensor, or compare the Doppler effect-based velocity measured through the array of microphones with the measurement value of the acceleration sensor. The electronic device (101) can measure the similarity using cosine similarity or Euclidean distance.

[0071] According to one embodiment, in operation 530, the electronic device (101) can determine the failure type of the first sensor. For example, the failure type can be classified into cases where the sensor is completely broken (Type 1), cases where the sensor data exceeds a set operating range (Type 2), and cases where it is recoverable due to a temporary malfunction (Type 3). The electronic device (101) can determine the failure type by using pattern analysis of the sensor data, statistical methods, and an anomaly detection algorithm. The failure types are merely examples and are not limited to three, and may vary depending on the settings.

[0072] According to one embodiment, the electronic device (101) can determine the type of sensor failure by classifying it into three types. The first type is when the sensor is completely broken, and a phenomenon may occur where data transmitted from the sensor driver becomes stuck. The second type is when the sensor data exceeds a set operating range, and the third type is when a temporary malfunction that may occur in an inertial sensor of a MEMS structure is recoverable. To determine these types of failure, the electronic device (101) may utilize outlier detection algorithms such as statistical methods, Z-scores, interquartile range (IQR), and Mahalanobis distance. Mahalanobis distance may refer to the distance from the center point (mean) of multivariate data to a specific data point. The electronic device (101) can calculate the Mahalanobis distance and detect outliers in the data by considering the correlation between variables. The electronic device (101) can simultaneously consider the scale of each variable and the correlation between variables using the data covariance matrix. The electronic device (101) can effectively identify abnormal patterns when multiple sensor data are collected simultaneously. The electronic device (101) can provide reliable outlier detection results in data that follows a normal distribution and can also utilize the Mahalanobis distance to detect anomalies in real-time multivariate time series data.

[0073] According to one embodiment, in operation 540, the electronic device (101) may select a processing method based on the determined failure type and generate input data for a prediction model. For the first type, the electronic device (101) may completely mask the corresponding sensor data or convert it to a preliminary model. For the second type, the electronic device (101) may correct the data through clipping or weight adjustment. For the third type, the electronic device (101) may apply interpolation based on previous data or window-based smoothing. The electronic device (101) may input the processed data into a multi-sensor-based prediction model and obtain reliable prediction results. The types of failure types and processing methods mentioned are merely examples and are not limited thereto. The types of failure types may vary depending on the classification, and the processing methods according to the failure types may also vary depending on the settings.

[0074] According to one embodiment, an electronic device (101) can estimate the acceleration of an object by measuring the Doppler effect of sound waves received through an array of microphones. The Doppler effect may refer to a change in frequency caused by the relative motion between a sound source and an observer. The electronic device (101) can extract velocity and acceleration information by analyzing the change in frequency. The electronic device (101) can determine whether the sensor is malfunctioning by comparing the estimated acceleration value with the measured value of the acceleration sensor.

[0075] According to one embodiment, the electronic device (101) can detect changes in the position of an object in consecutive video frames by utilizing optical flow or object tracking technology through a camera sensor. The electronic device (101) can determine that the sensor is faulty if the difference between the calculated velocity and acceleration values ​​and the sensor measurements exceeds a set threshold.

[0076] According to one embodiment, the electronic device (101) can detect an abnormality in the barometric pressure sensor by utilizing an acceleration sensor. The electronic device (101) can estimate a relative altitude change by double-integrating the acceleration in the vertical axis direction. The electronic device (101) can determine whether the barometric pressure sensor is malfunctioning by comparing the estimated relative altitude change with the altitude change measured by the barometric pressure sensor.

[0077] According to one embodiment, the electronic device (101) can determine whether there is constant velocity motion or linear motion by utilizing a buffer. The buffer may store position, time, velocity, and acceleration vector information at each point in time. The electronic device (101) can determine the motion state of an object by analyzing the information stored in the buffer. The electronic device (101) can measure changes in direction between consecutive acceleration vectors by utilizing cosine similarity.

[0078] According to one embodiment, the electronic device (101) can generate input data for a prediction model by selecting an appropriate processing method according to the determined fault type. For the first type, the electronic device (101) can add a sensor fault mask layer to completely mask the corresponding sensor data or convert it to a preliminary model. For the second type, the electronic device (101) can correct the data through clipping or weight adjustment. For the third type, the electronic device (101) can apply interpolation based on previous data or window-based smoothing.

[0079] According to one embodiment, the electronic device (101) can respond to sensor failures by preparing a plurality of backup models. The electronic device (101) can train each backup model by assuming a failure situation of a specific sensor. The electronic device (101) can maintain the accuracy of the system by switching to the corresponding backup model when an actual failure occurs.

[0080] According to one embodiment, the electronic device (101) can dynamically adjust the influence of sensor data through a variable mask method. The electronic device (101) can utilize the variable mask method in various situations including multimodal data processing, dynamic masking, and defect data supplementation. The electronic device (101) can flexibly adjust the prediction results of each model according to importance by assigning weights to the mask layer.

[0081] According to one embodiment, the electronic device (101) may attempt to recover from a stuck phenomenon that may occur in an inertial sensor of a MEMS structure by performing a power reset. A stuck phenomenon may refer to a state in which the vibrating mass or structure of the sensor is fixed in a specific position and does not move due to external shock, excessive acceleration, or static electricity. The recovery attempt may be performed within a set number of times (e.g., 10 times). The set number of times is merely an example and is not a fixed value, but may vary depending on the setting. The electronic device (101) may determine whether the recovery is successful by continuously monitoring the output of the sensor for each attempt.

[0082] FIG. 6 is a flowchart illustrating the operation of an electronic device according to one embodiment checking for abnormalities in acceleration data using a GPS sensor.

[0083] The operations described through FIG. 6 may be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (600) may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 4, and the technical features described above will be omitted below. The order of each operation in FIG. 6 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0084] According to one embodiment, operations 610 to 640 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0085] According to one embodiment, in operation 610, the electronic device (101) can receive data from sensors. The electronic device (101) can receive location data and time information from a GPS sensor. The electronic device (101) can receive 3-axis acceleration data from an acceleration sensor. The electronic device (101) can also collect information such as the number of connected satellites and GPS signal strength to determine the accuracy of the GPS signal.

[0086] According to one embodiment, in operation 620, the electronic device (101) can check conditions for determining whether there is an abnormality in the acceleration data. According to one embodiment, the electronic device (101) can receive location data and time information from a GPS sensor and receive 3-axis acceleration data from an acceleration sensor. The electronic device (101) can also collect information on the number of connected satellites and the signal strength of the GPS sensor to determine the accuracy of the GPS sensor signal. The electronic device (101) can check whether the accuracy of the received location data is greater than or equal to a preset reference value (e.g., horizontal accuracy within 5 meters) and can determine the straightness of the direction of travel calculated from continuous location data using cosine similarity. The reference value is merely an example and may vary depending on the setting. The electronic device (101) can determine whether the object (e.g., vehicle) is moving at a constant speed by checking whether the amount of change in continuous speed data is within a set threshold.

[0087] According to one embodiment, in operation 630, the electronic device (101) can measure the similarity between an acceleration value estimated from GPS data and a value measured by an actual acceleration sensor. The GPS-based acceleration estimation can be calculated by differentiating continuous position data twice. The electronic device (101) can calculate velocity from the time change of position and, in turn, calculate acceleration from the time change of velocity. The electronic device (101) can calculate the difference between the acceleration thus calculated and the acceleration measured by the sensor. If the calculated difference exceeds a set threshold value, the electronic device (101) can determine that there is a problem with the acceleration sensor.

[0088] According to one embodiment, in operation 640, the electronic device (101) can transmit the similarity measurement result to the prediction model. If an abnormality in the acceleration sensor is detected, the prediction model can mask the sensor data, adjust the weights, or replace it with an acceleration value calculated based on GPS. Through this, the electronic device (101) can maintain stable prediction performance even in the event of a sensor failure.

[0089] According to one embodiment, the electronic device (101) can check conditions for determining whether there is an acceleration anomaly using a GPS sensor. The electronic device (101) can check whether the accuracy of GPS speed information exceeds a specified level based on the strength of the GPS signal and the number of connected satellites. The electronic device (101) can check whether the object is in a straight-line driving state based on the directional analysis of continuous location data. The electronic device (101) can check whether the object is in a constant velocity motion state based on whether the amount of change in speed is within a set range.

[0090] According to one embodiment, the electronic device (101) can perform double differentiation to calculate the acceleration of an object from the position data of a GPS sensor. The electronic device (101) can calculate the velocity by dividing the change in continuous position data by time, and then calculate the acceleration by dividing the change in velocity by time.

[0091] According to one embodiment, the electronic device (101) may utilize various statistical methods to monitor sensor failures in real time. Outlier detection algorithms such as Z-score, interquartile range (IQR), and Mahalanobis distance may be applied, and the temporal continuity of the sensor data may also be analyzed.

[0092] According to one embodiment, the electronic device (101) may utilize GPS azimuth information to detect a failure of the gyroscope sensor. The GPS azimuth data may be reliable when the speed of the object is 10 kilometers per hour or more, and based on this, the normal operation of the sensor can be determined by comparing it with the angular velocity integral value of the gyroscope.

[0093] According to one embodiment, the electronic device (101) can disable the main sensor and replace it with a sub sensor when the sensor is completely broken. This allows for the backup of sensor data at the hardware level.

[0094] According to one embodiment, in the case of a recoverable failure, the electronic device (101) can monitor the state of the sensor for a specified time (e.g., 1 hour) to determine whether it returns to a normal range. The specified time is merely an example and is not limited thereto, and may vary depending on the settings. If the sensor does not return to a normal operating range within this time, the electronic device (101) can determine that it is a complete failure and can process extreme values ​​of the sensor data through non-linear scaling. The electronic device (101) can use a non-linear function such as tangent hyperbolic (tanh) or sigmoid to mitigate the impact of data exceeding the set operating range, thereby improving the stability of the model.

[0095] FIG. 7 is a flowchart illustrating the process of an electronic device according to one embodiment determining whether a gyro sensor is faulty.

[0096] The operations described through FIG. 7 may be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (700) may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 4, and the technical features described above will be omitted below. The order of each operation in FIG. 7 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0097] According to one embodiment, operations 702 to 722 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0098] According to one embodiment, in operation 702, the electronic device (101) can receive and process data in real time from at least one sensor. For example, the electronic device (101) can receive position, azimuth, and velocity data from a GPS sensor. Or the electronic device (101) can receive 3-axis angular velocity data from a gyroscope sensor. Angular velocity is a physical quantity representing the rotational motion speed of an object and may mean the amount of change in the angle of rotation per unit time.

[0099] According to one embodiment, in operation 710, the electronic device (101) can check whether the speed of the GPS is above a specified threshold (e.g., 10 km / h). The specified threshold is merely an example and is not limited thereto. The reliability of the GPS azimuth data may vary depending on the speed of the GPS. The GPS azimuth data may become more accurate when the object travels above a certain speed. The electronic device (101) can continue to monitor the sensor data when the speed is below the threshold (operation 702).

[0100] According to one embodiment, in operation 712, the electronic device (101) can determine similarity by utilizing GPS information. The electronic device (101) can compare the change in the GPS azimuth angle with the value obtained by integrating the angular velocity of the gyroscope sensor. Here, the integration of the angular velocity may mean calculating the change in direction over time. The electronic device (101) can perform a more accurate comparison using gyroscope data converted to an Earth coordinate system.

[0101] According to one embodiment, in operation 720, the electronic device (101) can determine whether the calculated similarity is below a set threshold. The electronic device (101) can determine that the measurement of the gyroscope sensor differs from the GPS-based change in direction based on the fact that the calculated similarity is lower than the threshold. If the calculated similarity exceeds the set threshold, the electronic device (101) can again determine the similarity by utilizing GPS information (712).

[0102] According to one embodiment, in operation 722, the electronic device (101) may determine that a gyroscope sensor exhibiting consistently low similarity is faulty. The electronic device (101) may perform sensor fault mask processing on the data of the sensor determined to be faulty, replace it with GPS azimuth data, or adjust the weights. Through such processing, the electronic device (101) can maintain stable direction estimation performance even in the event of a sensor failure and can continuously monitor the sensor data to check whether the faulty state persists.

[0103] FIG. 8 illustrates the process of reducing the influence of defect data by adding a mask layer to an electronic device according to one embodiment.

[0104] According to one embodiment, the electronic device (101) may implement a mask layer structure for handling sensor failures in a multi-sensor-based prediction model as shown in FIG. 8. Here, the mask layer may play a role in improving the reliability of the model by controlling the influence of defective sensor data. The electronic device (101) may process each sensor data through individual models; specifically, the accelerometer model (810) may process data from the accelerometer sensor, the gyroscope model (820) may process data from the gyroscope sensor, and other sensor models (830) may process data from other sensors such as a barometer. Each model may have a structure optimized for the characteristics of the corresponding sensor. The electronic device (101) may use the mask layer to extract features of the sensor data and learn meaningful patterns.

[0105] According to one embodiment, the electronic device (101) can control the influence of data according to the failure state of each sensor through sensor fault mask layers (812, 822, 832). For example, the electronic device (101) can completely block data by setting the corresponding mask value to 0 when the sensor is completely broken (Type 1). The electronic device (101) can reduce the influence of data by setting the mask value to a value between 0 and 1 when the sensor is partially faulty (Type 2). The electronic device (101) can utilize the data as is by setting the mask value to 1 for a sensor that is operating normally.

[0106] According to one embodiment, the electronic device (101) can integrate the outputs of each sensor model that have been masked through a fusion layer (840). The fusion layer can combine the outputs of each sensor model by means of weighted sum or concatenation to generate a single integrated feature vector. In this case, the influence of each sensor model's output on the final result can be automatically adjusted according to the mask value.

[0107] According to one embodiment, the electronic device (101) can derive a final prediction result through a classification layer (850). The classification layer can receive an integrated feature vector generated from a fusion layer as input and perform a desired prediction task (e.g., collision detection, activity classification). The electronic device (101) can maintain stable prediction performance even in the event of sensor failure through a hierarchical structure. The electronic device (101) can derive relatively reliable results by adaptively processing data according to the state of each sensor.

[0108] FIG. 9 illustrates the process of an electronic device according to one embodiment improving the accuracy of a model by failure type.

[0109] The operations described through FIG. 9 may be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (700) may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 4, and the technical features described above will be omitted below. The order of each operation in FIG. 9 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0110] According to one embodiment, operations 902 to 932 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0111] According to one embodiment, the electronic device (101) can improve the accuracy of the model by applying different processing methods depending on the type of sensor failure. In operation 902, the electronic device (101) can detect a sensor failure. The electronic device (101) can detect a sensor failure through any one of statistical analysis of sensor data, an outlier detection algorithm, or a comparison of similarity with other sensors. For example, the electronic device (101) can detect abnormal patterns in the data using a Z-score or interquartile range (IQR) method, or analyze the correlation between GPS data and acceleration sensor data.

[0112] According to one embodiment, in operation 904, the electronic device (101) can classify the detected failure into each type (e.g., three types) based on the severity and recoverability of the failure. The electronic device (101) can determine the type of failure by comprehensively analyzing the pattern of change in sensor data, the duration of the abnormal value, and the degree of deviation from the normal range.

[0113] According to one embodiment, the first type (910) may represent a complete failure state. In this case, the electronic device (101) may, in operation 912, process the corresponding sensor data using a sensor fault mask or switch to a pre-prepared backup model. The backup model may be a model separately trained to use only the data of the remaining sensors excluding the faulty sensor.

[0114] According to one embodiment, the second type (920) may indicate a case where sensor data exceeds the normal operating range. In this case, the electronic device (101) may apply at least one of weight reduction, clipping, or non-linear scaling in operation 922. Specifically, the electronic device (101) may dynamically assign a weight between 0 and 1 depending on the degree to which the data deviates from the normal range, or use a tangent hyperbolic (tanh) function to mitigate the effect of extreme values.

[0115] According to one embodiment, the third type (930) may represent a case where recovery is possible due to a temporary malfunction. In this case, the electronic device (101) may process the data in operation 932 using at least one of smoothing, interpolation, or variable weighting. For example, the electronic device (101) may remove temporary noise using a moving average filter or interpolate the data of the defective section using values ​​of the normal data section. Additionally, the electronic device (101) may reflect the sensor returning to a normal state by gradually adjusting the weights over time.

[0116] According to one embodiment, the electronic device (101) can receive and process sensing data in real time from various sensors. The electronic device (101) may include a portable electronic device such as a smartphone or a smartwatch. The electronic device (101) may include a plurality of sensors. The plurality of sensors may include, for example, an accelerometer, a gyroscope, a GPS sensor, a microphone, or a camera. The types of sensors are merely examples and are not limited thereto.

[0117] According to one embodiment, the electronic device (101) can check in real time whether the first sensor is faulty based on the sensing data of the second sensor among a plurality of sensors. The operation of checking whether the sensor is faulty may mean an operation of evaluating the reliability of data received from the sensor. The electronic device (101) can determine whether the first sensor is faulty by comparing and analyzing the similarity between the sensing data of the first sensor and the second sensor.

[0118] According to one embodiment, when a failure of the first sensor is confirmed, the electronic device (101) can determine the type of failure. The failure types can be broadly classified into three types. The electronic device (101) can determine the first type as a case where no data is received from the sensor driver or there is no change in the data value. The electronic device (101) can determine the second type as a case where the sensor data exceeds a set operating range. The electronic device (101) can determine the third type as a case where the sensor data can return to a normal operating range within a set time or a number of power resets.

[0119] According to one embodiment, the electronic device (101) may apply different processing methods depending on the determined failure type. In the case of the first type, the electronic device (101) may perform masking processing to completely exclude the data of the corresponding sensor, or process by switching to a pre-trained preliminary model. In the case of the second type, the electronic device (101) may perform clipping processing to limit the sensing data to within a set normal operating range, or process by reducing the weight of the data. In the case of the third type, the electronic device (101) may perform interpolation processing to estimate the data at the current time point based on the normal sensing data from the previous time point, or perform window-based smoothing processing using data within a set time interval.

[0120] According to one embodiment, the electronic device (101) can detect whether the first sensor is malfunctioning by utilizing a GPS sensor. Specifically, the velocity and acceleration of an object can be calculated by analyzing the temporal change in location data continuously received from the GPS sensor. Here, velocity can be calculated as the temporal change in location, and acceleration can be calculated as the temporal change in velocity. The electronic device (101) calculates the difference between the acceleration thus calculated and the sensing data of the first sensor (e.g., acceleration sensor), and if this difference exceeds a set threshold, it can determine that the first sensor is malfunctioning.

[0121] According to one embodiment, the electronic device (101) can check specific conditions to increase the accuracy of fault detection using a GPS sensor. The electronic device (101) can check whether the accuracy of location data received from the GPS sensor is greater than or equal to a preset reference value. The electronic device (101) can determine the accuracy of the location data based on the strength of the GPS signal or the number of connected satellites. The electronic device (101) can check whether the direction of travel calculated from continuous location data is a straight line. The electronic device (101) can check whether the direction of travel is a straight line by analyzing the angle change between direction vectors. The electronic device (101) can check whether the amount of change in the calculated speed is within a preset constant speed determination range.

[0122] According to one embodiment, the electronic device (101) can store and analyze changes in sensing data over time using memory. The electronic device (101) can determine changes in direction by calculating the cosine similarity between consecutive sensing data stored in memory. Cosine similarity is a similarity measurement method based on the angle between two vectors, and a value closer to 1 indicates higher similarity. The electronic device (101) can determine whether the sensor is malfunctioning through the analysis of changes in direction.

[0123] According to one embodiment, the electronic device (101) can detect whether the sensor is malfunctioning through a microphone. The electronic device (101) can measure the Doppler effect on sound waves received through an array of multiple microphones. The Doppler effect may refer to a change in frequency caused by relative motion between a sound source and an observer. The electronic device (101) can estimate the speed and acceleration of an object by analyzing the Doppler effect. The electronic device (101) can determine whether the sensor is malfunctioning by comparing the estimated speed and acceleration with the sensing data of the first sensor.

[0124] According to one embodiment, the electronic device (101) can determine whether the sensor is malfunctioning by analyzing consecutive video frames received through a camera sensor. The electronic device (101) can determine the positional change of a specific object (e.g., a fixed object or lane on a road) in the consecutive video frames. Through the temporal analysis of this positional change, the relative speed and acceleration of the object can be estimated, and the malfunction can be determined by comparing this with the sensing data of the first sensor.

[0125] According to one embodiment, the electronic device (101) can determine whether the sensor is faulty through mutual verification between the gyroscope and the accelerometer. The change in direction can be calculated by integrating the angular velocity data received from the gyroscope over time, and this can be compared with the change in direction calculated from the sensing data of the accelerometer. If the difference between the change in direction calculated from the two sensors exceeds a set threshold, it can be determined that the sensor is faulty.

[0126] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0127] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0128] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0129] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0130] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0131] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, Memory that stores instructions and includes one or more storage media; It includes at least one processor comprising processing circuitry, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device Receiving sensing data from multiple sensors, and The similarity with the sensing data of the second sensor among the plurality of sensors is compared and analyzed in real time to determine whether the first sensor is faulty, and Based on the confirmation that the failure of the first sensor is confirmed, the type of failure of the first sensor is determined, and Based on the above-determined failure type, at least one processing of masking, weight adjustment, or interpolation is performed on the sensing data of the first sensor to generate input data for a multi-sensor-based prediction model, and An electronic device that controls the inference of the prediction model using the input data generated above.

2. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Based on the case where data is not received from the sensor driver or the data value does not change, the failure type of the first sensor is determined as the first type, and Based on the fact that the sensor data exceeds a set operating range, the failure type of the first sensor is determined as the second type, and An electronic device that controls the determination of the failure type of the first sensor as a third type based on whether the sensor data can return to a set operating range within a set time or the number of power resets.

3. In Paragraph 2, When the above instructions are executed by the at least one processor, the electronic device, An electronic device that controls the sensing data of the first sensor to be masked or converted to a pre-learned preliminary model for processing, based on the failure type of the first sensor being determined to be the first type.

4. In Paragraph 2, When the above instructions are executed by the at least one processor, the electronic device, An electronic device that controls the sensing data of the first sensor to be limited to a set normal operating range by performing clipping processing based on the failure type of the first sensor being determined to be the second type, or to process by reducing the weight set for the sensing data.

5. In Paragraph 2, When the above instructions are executed by the at least one processor, the electronic device, An electronic device that controls the estimation of current sensing data based on normal sensing data from a previous time point, or the estimation of current sensing data based on the failure type of the first sensor being determined to be the third type, or controls the estimation of current sensing data based on normal sensing data from a previous time point, or controls the estimation of window-based smoothing processing using sensing data within a set time interval.

6. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, The accuracy of the location data received from the GPS sensor is greater than or equal to a preset threshold value, and The direction of travel calculated from continuous location data is a straight line, and If the change in the calculated speed is within a preset constant speed determination range, it is determined to check whether the first sensor is faulty. Based on the determination of whether the first sensor is faulty, the speed and acceleration of the vehicle are calculated from the temporal change amount of position data continuously received from the second sensor, and An electronic device that controls the first sensor to check for failure based on the difference value between the calculated acceleration and the acceleration sensing data of the first sensor.

7. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Based on the failure type of the first sensor above An electronic device that controls displaying a notification indicating whether to replace the first sensor.

8. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Using the above memory, changes in sensing data over time are stored, and The change in direction is confirmed by calculating the cosine similarity between the sensing data stored in the above memory, and An electronic device that controls the first sensor to check for failure based on the above change in direction.

9. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Estimating the speed and acceleration of a vehicle by measuring the Doppler effect of sound waves received through a microphone array including a plurality of microphones as the second sensor, and An electronic device that controls the first sensor to check for failure based on the difference between the estimated acceleration and the acceleration sensing data of the first sensor.

10. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Tracking changes in the position of a specific object in consecutive video frames received through a camera as the second sensor, and Estimating the velocity and acceleration of the object from the tracked position change, An electronic device that controls the first sensor to check for failure based on the difference between the estimated acceleration and the acceleration sensing data of the first sensor.

11. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, In the case where the first sensor is an accelerometer and the second sensor is a gyroscope, The amount of change in direction is calculated by integrating the angular velocity data received from the above gyroscope over time, and An electronic device that controls the detection of whether the first sensor is faulty based on the difference between the calculated change in direction and the change in direction calculated from the sensing data of the acceleration sensor.

12. A computer-readable non-transient storage medium storing one or more programs comprising instructions executable by a processor of an electronic device, When the above instructions are executed, the electronic device, Receiving sensing data from multiple sensors, and The similarity with the sensing data of the second sensor among the plurality of sensors is compared and analyzed in real time to determine whether the first sensor is faulty, and Based on the confirmation that the failure of the first sensor is confirmed, the type of failure of the first sensor is determined, and Based on the above-determined failure type, at least one processing of masking, weight adjustment, or interpolation is performed on the sensing data of the first sensor to generate input data for a multi-sensor-based prediction model, and A computer-readable non-transient storage medium that enables inference of the prediction model using the input data generated above.

13. In Paragraph 12, When the above instructions are executed, the electronic device, Based on the case where data is not received from the sensor driver or the data value does not change, the failure type of the first sensor is determined as the first type, and Based on the fact that the sensor data exceeds a set operating range, the failure type of the first sensor is determined as the second type, and A computer-readable non-transient storage medium that determines the failure type of the first sensor as a third type based on the fact that the sensor data can return to a set operating range within a set time or a number of power resets.

14. In Paragraph 13, Based on the fact that the failure type of the first sensor is determined to be the first type, control is made to mask the sensing data of the first sensor or to convert it to a pre-trained preliminary model for processing, and A computer-readable non-transient storage medium that controls the processing to limit the sensing data of the first sensor to within a set normal operating range based on the failure type of the first sensor being determined to be the second type, or to reduce the weight set for the sensing data.

15. In a method of operating an electronic device, Operation of receiving sensing data from multiple sensors, An operation to check whether the first sensor is faulty by comparing and analyzing the similarity with the sensing data of the second sensor among multiple sensors in real time, An operation to determine the type of failure of the first sensor based on confirmation that the failure of the first sensor is confirmed, The operation of generating input data for a multi-sensor-based prediction model by performing at least one of masking, weight adjustment, or interpolation on the sensing data of the first sensor based on a determined failure type, and A method comprising the operation of performing inference of the prediction model using generated input data.

Images