On-device ai inference optimization method based on multiple bio-signals and device thereof
The method optimizes NPU-based neural network inference by differentiating memory layers and performing quantization based on biosignal characteristics, enhancing accuracy and efficiency in processing multiple biosignals.
Patent Information
- Application Number
- PCT/KR2026/002993
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-02-23
- Publication Date
- 2026-11-12
Smart Images

Figure KR2026002993_12112026_PF_FP_ABST
Abstract
Description
Multi-biometric signal-based on-device AI inference optimization method and apparatus
[0001] The present invention relates to a method and apparatus for optimizing on-device AI inference based on multiple biosignals, and more specifically, to a method and apparatus for optimizing NPU-based neural network inference by considering the interaction between multiple biosignals.
[0002] This invention is a study conducted by the Korea Electronics Technology Institute under the research project title "Development and Demonstration of a Domestic SoC-based On-Device AI Hyper-Personalized Home Appliance System" for the "Industrial Core Technology Development Project" with support from the Ministry of Trade, Industry and Energy (Project No. RS-2024-00507555).
[0003] Recently, personalized smart healthcare services and wearable devices based on biosignals have been gaining popularity, and accordingly, the importance of technology for collecting and analyzing various biosignal data in real time is increasing.
[0004] Various biosignals generally include electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), gelatinized saturation (GSR), and photoplethysmography (PPG), and these biosignals have different frequency characteristics, noise patterns, measurement locations, and sensitivities.
[0005] Meanwhile, conventional technologies for collecting and analyzing various biosignal data have limitations in that they process each of these diverse biosignals individually, failing to reflect the correlations between them and thus unable to consider the non-linear relationships between signals. Furthermore, since all biosignal data is processed in the same manner, there are many unnecessary calculations, leading to reduced processing speed and increased power consumption. Additionally, compressing various biosignals in the same way can result in the loss of important information in some signals.
[0006] Accordingly, there is a growing need for technology to increase real-time inference speed in on-device environments by considering the interactions between various biosignals and the characteristics of each signal.
[0007] The objective of the present invention is to provide a multi-biosignal-based on-device AI inference optimization method and apparatus that optimize NPU-based neural network inference by considering the interaction between multiple biosignals.
[0008] A method for optimizing on-device AI inference based on multiple biosignals according to an embodiment of the present invention for achieving such objectives comprises: a step of collecting multiple biosignals; a step of performing time-series data preprocessing and Cross-GAF transformation on the multiple biosignals; a step of allocating computations by differentiating memory layers within a Neural Processing Unit (NPU) according to the importance of each of the multiple biosignal characteristics; a step of performing quantization corresponding to each of the multiple biosignal characteristics; and a step of analyzing the multiple biosignals through a neural network model optimized through the memory layer differentiation and quantization to output an inference result.
[0009] Here, the multiple biosignals may include at least one of an electrocardiogram signal, a ballistocardiogram signal, an electroencephalogram signal, an electromyogram signal, a skin conduction signal, a photoplethysmogram signal, and a phonocardiogram signal.
[0010] In addition, the step of performing the time series data preprocessing and Cross-GAF transformation may include analyzing the power spectral density of each of the multiple biosignals to perform filtering corresponding to each of the characteristics of the multiple biosignals, performing normalization corresponding to the amplitude characteristics of each of the multiple biosignals, performing synchronization between the multiple biosignals, and optimizing the quality of the multiple biosignals.
[0011] In addition, the step of performing the time series data preprocessing and Cross-GAF conversion can convert the interaction between the multiple biosignals into a 2D image form.
[0012] In addition, the step of allocating computations by differentiating memory hierarchy within the NPU according to the importance of each of the multiple biosignal characteristics can optimize memory within the NPU through at least one of a method of allocating memory by analyzing the information entropy of the multiple biosignals, a method of performing memory prefetching based on the periodic characteristics of each of the multiple biosignals, a method of allocating memory based on the diagnostic priority of the multiple biosignals, and a method of allocating memory by considering the correlation between the multiple biosignals.
[0013] In addition, the step of performing quantization corresponding to each of the multiple biosignal characteristics may be performed through at least one of a method of selecting a quantization bit width based on the clinical importance and frequency characteristics of each of the multiple biosignals, a method of selecting a bit width corresponding to the sensitivity of each of the multiple biosignals, and a method of selecting a bit width corresponding to a specific frequency band of each of the multiple biosignals.
[0014] Meanwhile, a computer-readable recording medium according to one embodiment of the present invention may include a program for executing a multi-biometric signal-based on-device AI inference optimization method on a computer.
[0015] Meanwhile, a multi-biometric signal-based on-device AI inference optimization device according to one embodiment of the present invention includes a sensor unit for collecting multi-biometric signals and a processor that performs time-series data preprocessing and Cross-GAF conversion on the collected multi-biometric signals, differentiates memory layers within an NPU according to the importance of each of the multi-biometric signal characteristics to allocate computations, performs quantization corresponding to each of the multi-biometric signal characteristics, and analyzes the multi-biometric signals through a neural network model optimized through the memory measurement differentiation and quantization to output an inference result.
[0016] Here, the multiple biosignals may include at least one of an electrocardiogram signal, a ballistocardiogram signal, an electroencephalogram signal, an electromyogram signal, a skin conduction signal, a photoplethysmogram signal, and a phonocardiogram signal.
[0017] In addition, the processor can analyze the power spectral density of each of the multiple biosignals to perform filtering corresponding to each of the characteristics of the multiple biosignals, perform normalization corresponding to the amplitude characteristics of each of the multiple biosignals, perform synchronization between the multiple biosignals, and perform quality optimization of the multiple biosignals.
[0018] In addition, the processor can convert the interaction between the multiple biosignals into a 2D image form.
[0019] In addition, the processor can optimize memory within the NPU through at least one of the following methods: a method of allocating memory by analyzing the information entropy of the multiple biosignals, a method of performing memory prefetching based on the periodic characteristics of each of the multiple biosignals, a method of allocating memory based on the diagnostic priority of the multiple biosignals, and a method of allocating memory by considering the correlation between the multiple biosignals.
[0020] According to various embodiments of the present invention as described above, by reflecting the characteristics of various biosignal data, analysis considering the interaction between signals is possible, thereby improving inference accuracy and maximizing computational efficiency and minimizing power consumption, thus ensuring high accuracy and computational efficiency in an NPU-based on-device AI environment.
[0021] FIG. 1 is a flowchart illustrating a multi-biometric signal-based on-device AI inference optimization method according to one embodiment of the present invention.
[0022] FIG. 2 is a diagram illustrating an example of customized filtering according to biosignal characteristics according to one embodiment of the present invention.
[0023] FIG. 3 is a diagram illustrating an example of SNR-based biosignal quality optimization according to one embodiment of the present invention.
[0024] FIGS. 4 and FIGS. 5 are drawings illustrating examples of Cross-GAF conversion according to an embodiment of the present invention.
[0025] FIG. 6 is a diagram illustrating a biosignal information entropy-based memory allocation process according to one embodiment of the present invention.
[0026] FIG. 7 is a diagram illustrating an example of memory allocation of ECG signals by entropy according to one embodiment of the present invention.
[0027] FIG. 8 is a diagram illustrating multilayer memory management based on biosignal diagnosis priority according to one embodiment of the present invention.
[0028] FIG. 9 is a diagram illustrating a biosignal specialized quantization technique according to one embodiment of the present invention.
[0029] FIG. 10 is a diagram illustrating precision-preserving quantization by frequency band according to one embodiment of the present invention.
[0030] FIG. 11 is a block diagram illustrating the configuration of a multi-biometric signal-based on-device AI inference optimization device according to one embodiment of the present invention.
[0031] FIG. 12 is a block diagram illustrating the specific configuration of the multi-biometric signal-based on-device AI inference optimization device illustrated in FIG. 11.
[0032] FIG. 13 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention.
[0033] The present invention will be described in more detail below with reference to the drawings. Furthermore, in describing the present invention, detailed descriptions of related known functions or configurations are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the invention. Additionally, the terms described below are defined considering their functions in the present invention, and these may vary depending on the intentions or relationships of the user or operator. Therefore, their definitions should be based on the content throughout this specification.
[0034] FIG. 1 is a flowchart illustrating a multi-biometric signal-based on-device AI inference optimization method according to one embodiment of the present invention.
[0035] Referring to FIG. 1, a method for optimizing on-device AI inference based on multiple biosignals according to an embodiment of the present invention may include the steps of: collecting multiple biosignals (S110); performing time-series data preprocessing and Cross-GAF conversion on the multiple biosignals (S120); differentiating memory layers within a Neural Processing Unit (NPU) according to the importance of each of the multiple biosignal characteristics and allocating computations (S130); performing quantization corresponding to each of the multiple biosignal characteristics (S140); and analyzing the multiple biosignals through a neural network model optimized through memory layer differentiation and quantization to output an inference result (S150).
[0036] Herein, a multi-biometric signal-based on-device AI inference optimization method according to one embodiment of the present invention relates to a method for optimizing NPU-based AI inference by reflecting the interaction between multiple biometric signals (ECG (electrocardiogram), BCG (ballistocardiogram), EEG (brainwave), EMG (electromyogram), GSR (skin conduction), PPG (photoplethysmography), SCG (phonocardiogram), PCG (heart sound), Resp (respiratory signal), etc.). This multi-biometric signal-based on-device AI inference optimization method may be implemented by being installed in the form of firmware, programs, or embedded software on a chip, processor, or system-on-chip (SoC) embedded within a multi-biometric signal-based on-device AI inference optimization device, or by being implemented in the form of software or programs downloadable within the cloud.
[0037] Meanwhile, the on-device AI inference optimization method based on multiple biosignals according to one embodiment of the present invention can be broadly distinguished into a data preprocessing process and an NPU optimization process, wherein the data preprocessing process includes a step (S120) of performing time-series data preprocessing and Cross-GAF conversion on multiple biosignals, and the NPU optimization process may include a step (S130) of assigning computations by differentiating memory layers within the NPU (Neural Processing Unit) according to the importance of each of the multiple biosignal characteristics, and a step (S140) of performing quantization corresponding to each of the multiple biosignal characteristics.
[0038] Here, the data preprocessing process may consist of time-series data normalization, signal quality improvement, time synchronization between multiple biosignals, and Cross-GAF transformation. This data preprocessing process can be utilized in both training and inference and can be performed before being converted into input data for a neural network model.
[0039] In particular, noise removal, normalization, and correlation analysis between biosignals can be applied to improve biosignal quality.
[0040] Furthermore, the NPU optimization process includes memory metrology, biosignal-specific quantization, and computational reduction, and can be designed to efficiently process multiple biosignal data. This optimization process optimizes computational resources and reduces the size of the neuro-speech inference model while maintaining accuracy; additionally, to achieve model lightweighting and real-time data processing, a method can be utilized that analyzes the importance of biosignals and adjusts and reflects their weights.
[0041] Meanwhile, the step of collecting multiple biosignals (S110) can be collected through various sensors, wherein the multiple biosignals may include at least one of an electrocardiogram signal, a ballistocardiogram signal, a brainwave signal, an electromyogram signal, a skin conduction signal, a photoplethysmogram signal, and a phonocardiogram signal.
[0042] In particular, since biosignals have the characteristics of time-series data, synchronization is essential during the acquisition phase. Because various biosignals have different sampling rates and noise characteristics, time alignment between biosignals must be performed during the acquisition phase. To this end, timestamp-based synchronization and linear interpolation can be applied to correct sample alignment between biosignal data.
[0043] Specifically, the collected biosignal can be expressed as shown in Equation 1 below.
[0044]
[0045] Here, refers to the collected biosignals, and represents a periodic signal, and means a non-periodic signal.
[0046] In addition, sample correction between biosignal data can be performed through timestamp-based synchronization and linear interpolation for periodic and non-periodic signals distinguished from the biosignals.
[0047] In addition, stable data input can be ensured by applying baseline adjustment techniques to minimize the impact caused by sensor drift and external environmental factors during the collection phase.
[0048] In addition, the step (S120) of performing time-series data preprocessing and Cross-GAF transformation on multiple biosignals is a step of converting the interaction between multiple biosignals into a 2D image form and optimizing it as input to a CNN-based neural network model, which can effectively capture phase relationships and amplitude patterns between biosignals.
[0049] In addition, the step (S120) of performing time-series data preprocessing and Cross-GAF transformation on multiple biosignals can analyze the power spectral density of each of the multiple biosignals to perform filtering corresponding to each of the multiple biosignals, perform normalization corresponding to the amplitude characteristics of each of the multiple biosignals, perform synchronization between the multiple biosignals, and perform quality optimization of the multiple biosignals.
[0050] Specifically, the step (S120) of performing time-series data preprocessing and Cross-GAF conversion on multiple biosignals can perform customized filtering by automatically selecting optimal filter parameters according to the characteristics of each biosignal, and to this end, filtering can be performed by analyzing the power spectral density of each of the multiple biosignals.
[0051] For example, the spectral density of a fractal signal constituting part of a biosignal can be expressed as shown in Equation 2 below.
[0052]
[0053] Here, since β is generally about 1.5 for EEG (brainwave) signals and β is generally about 1 for ECG (electrocardiogram) signals, the range of β was set to a range of 0 to 2.
[0054] In addition, the filtering frequency band for each biosignal can be set as follows.
[0055] For example, a ballistocardiogram (BCG) signal containing heart rate, respiration, and sleep information can be set to a 0.1-60Hz bandpass filter, an electroencephalogram (EEG) signal containing delta, theta, alpha, beta, and gamma waves can be set to a 0.5-45Hz bandpass filter, an electrocardiogram (ECG) signal containing heart rate information can be set to a 0.5-40Hz bandpass filter, and an electromyogram (EMG) signal containing muscle activity can be set to a 20-450Hz bandpass filter.
[0056] In addition, the skin conductivity (GSR) signal including changes in electrical conductivity can be set to a 0.05-10Hz low-pass filter, the photoplethysmography (PPG) signal including changes in heart rate and blood flow can be set to a 0.5-8Hz band-pass filter, the phonogram (SCG) signal including heart vibration information can be set to a 0.6-25Hz band-pass filter, and the phonogram (PCG) signal including S1 and S2 heart sound characteristics can be set to a 20-200Hz band-pass filter.
[0057] Of course, the filtering frequency bands for each biosignal as described above may be set differently depending on factors such as the classification learning model, analysis goal, and surrounding noise environment.
[0058] Customized filtering can be performed for each different biosignal through the frequency bandpass filter configured in this way.
[0059] FIG. 2 is a diagram illustrating an example of customized filtering according to biosignal characteristics according to one embodiment of the present invention.
[0060] Referring to FIG. 2, a brainwave signal (110) before filtering and a brainwave signal (120) after filtering are shown, and a 0.5-45Hz bandpass filter was used for filtering the brainwave signal.
[0061] Meanwhile, the step (S120) of performing time-series data preprocessing and Cross-GAF transformation on multiple biosignals can perform normalization corresponding to the amplitude characteristics of each of the multiple biosignals and can correct discrepancies between multiple biosignal data by performing synchronization between the multiple biosignals.
[0062] To this end, cross-correlation analysis is utilized to establish a reference signal, and time delays are corrected so that all biosignals can be aligned on the same time axis.
[0063] Specifically, the following methods can be used in the normalization process.
[0064]
[0065] (1) Min-Max Normalization
[0066] Min-Max normalization is a method of correcting relative magnitude differences by adjusting the maximum and minimum values of a biosignal to a range of 0 to 1 as shown in Equation 3 below, and can be used mainly when the scale of the signal is constant.
[0067]
[0068]
[0069] (2) Z-score normalization
[0070] Z-score normalization is the average of the biosignal as shown in Equation 4 below. and standard deviation This is a method of converting to a standard normal distribution using [this method], and is effective for biological signals that mainly follow a normal distribution, such as brainwave signals or electrocardiogram signals.
[0071]
[0072]
[0073] (3) Robust Normalization
[0074] Robust normalization is a method that performs robust normalization against outliers by transforming based on the median and interquartiles (IQR) as shown in Equation 5 below, and is used to reduce the influence of outliers.
[0075]
[0076]
[0077] In addition, for synchronization between multiple biosignals, precise synchronization is performed between biosignals with different sampling rates.
[0078] Specifically, the biosignal with the highest sampling rate is set as the reference to perform synchronization with other biosignals, and here, the time difference of each biosignal is analyzed to calculate the time difference requiring correction through the following mathematical formula 6.
[0079]
[0080] Here, biosignals and reference signal By analyzing the correlation between them, the time difference can be calculated, and the one with the highest correlation value Time delay can be corrected based on the value.
[0081] In addition, linear interpolation and spline interpolation can be utilized to align biosignals with time differences to a reference signal. Linear interpolation is performed using Equation 7 below, and spline interpolation can be performed using Equation 8 below.
[0082]
[0083]
[0084] Here, the interpolation coefficients a, b, c, and d can be optimized for each biosignal.
[0085] And as described above, an integrated data frame can be generated so that the time-delay-corrected biosignals are aligned on the same time axis.
[0086] Meanwhile, the step (S120) of performing time series data preprocessing and Cross-GAF conversion can optimize the quality of multiple biosignals, specifically by calculating a quality score for selecting optimal data and including only reliable data in the Cross-GAF conversion process based on this.
[0087] Here, the process for calculating the quality score is as follows. First, a minimum SNR threshold can be set for each biosignal.
[0088] SNR can be calculated using the following mathematical formula 9.
[0089]
[0090] In addition, the minimum SNR threshold for each biosignal can be set as follows.
[0091] Due to the characteristics of measuring minute body vibrations, the minimum SNR threshold for the ballistocardiogram (BCG) signal can be set to 14 dB; due to the characteristics of low amplitude and complex frequency, the minimum SNR threshold for the electroencephalogram (EEG) signal can be set to 10 dB; considering the need for accurate identification of the QRS complex, the minimum SNR threshold for the electrocardiogram (ECG) signal can be set to 18 dB; due to the characteristics of signal intensity change during muscle contraction, the minimum SNR threshold for the electromyogram (EMG) signal can be set to 12 dB; due to the characteristics of slow change and low amplitude, the minimum SNR threshold for the skin conductance (GSR) signal can be set to 14 dB; due to the need for accurate waveform preservation of the pulse wave, the minimum SNR threshold for the photoplethysmogram (PPG) signal can be set to 16 dB; due to the characteristics of cardiac mechanical activity, the minimum SNR threshold for the phonocardiogram (SCG) signal can be set to 15 dB; and considering the need for accurate identification of heart sounds, the minimum SNR threshold for the phonocardiogram (PCG) signal can be set to 16 dB.
[0092] Accordingly, biosignal segments in which the SNR calculated through Equation 9 is lower than the minimum SNR threshold of each biosignal are automatically excluded, and only biosignal segments with an SNR higher than the minimum SNR threshold are selected so that only data maintaining the best quality can be used in the Cross-GAF conversion and NPU inference processes.
[0093] In addition, the positive / negative ratio of the periodic components can be calculated using the following mathematical formula 10, and the balance of the signal can be analyzed to correct data distortion.
[0094]
[0095] SI: Symmetry Index
[0096]
[0097] In addition, through quantitative analysis of biosignal data intervals, it is possible to evaluate whether the biosignal maintains a constant quality. For example, the coefficient of variation (CV) is analyzed after dividing the interval into 2-second intervals, and can be calculated using CV = σ / μ. If the CV is 0.35 or less, it can be evaluated as being within the normal range.
[0098] Through this process, signal segments below the minimum SNR threshold for each biosignal are automatically excluded.
[0099] Of course, these signal quality standards can be adjusted according to various factors, such as the classification learning model environment, analysis objectives, ambient noise environment, measurement conditions, subject characteristics (age, health status), and the performance of the measurement equipment.
[0100] FIG. 3 is a diagram illustrating an example of SNR-based biosignal quality optimization according to one embodiment of the present invention.
[0101] Referring to FIG. 3, high-quality signal (210), medium-quality signal (220), and low-quality signal (230) are shown, and it can be seen that the SNR of the high-quality signal (210) is the highest, and in the waveform shape of the signal, it can be seen that it contains only the necessary data without noise.
[0102] Meanwhile, the step (S120) of performing time series data preprocessing and Cross-GAF transformation can convert the interaction between multiple biosignals into a 2D image form, and instead of the existing GAF / RP transformation for processing a single signal, a Cross-GAF transformation that effectively reflects the interaction between multiple signals is applied, and can be transformed through the following mathematical formula 11.
[0103]
[0104]
[0105] Here, means different biosignals, and represents the average value of each biosignal, and arctan2 is a two-factor tangent function that can calculate angles in all quadrants.
[0106] This Cross-GAF transformation can effectively preserve the phase relationship between multiple signals and simultaneously captures temporal patterns and correlations between signals to fuse and analyze the two signals, making it effective for expressing complex non-linear relationships and optimal for CNN-based models because it is converted into 2D image data.
[0107] FIGS. 4 and FIGS. 5 are drawings illustrating examples of Cross-GAF conversion according to an embodiment of the present invention.
[0108] Referring to FIG. 4, an electrocardiogram (ECG) signal (310) and an electroencephalogram (EEG) signal (320) are shown, and the result (330) of a Cross-GAF conversion performed on the electrocardiogram (ECG) signal (310) and the electroencephalogram (EEG) signal (320) is shown.
[0109] Here, referring to the result (330) of the Cross-GAF transformation, the X-axis represents the time order of the brainwave signal and the Y-axis represents the time order of the electrocardiogram signal, and the color value represents the cosine mutual similarity value and is expressed in the range from -1.0 (purple) to 1.0 (yellow).
[0110] In particular, when referring to the result (330) of the Cross-GAF conversion, strong periodicity is observed in each of the electrocardiogram (ECG) signal (310) and the brainwave (EEG) signal (320), indicating that the base rhythm of the biosignal has been stably measured, and in the area marked in yellow, it can be seen that the directionality of the electrocardiogram (ECG) signal (310) and the brainwave (EEG) signal (320) rising or falling simultaneously is synchronized.
[0111] Also, referring to FIG. 5, an electromyogram (EMG) signal (410) and a skin conductance (GSR) signal (420) are shown, and the result (430) of a Cross-GAF conversion performed on the electromyogram (EMG) signal (410) and the skin conductance (GSR) signal (420) is shown.
[0112] Here, referring to the result (430) of the Cross-GAF transformation, the X-axis represents the time order of the skin conductivity signal and the Y-axis represents the time order of the electromyography signal, and the color values represent the cosine mutual similarity values and are expressed in the range from -1.0 (purple) to 1.0 (yellow).
[0113] In particular, referring to the result (430) of the Cross-GAF conversion, it is evident that strong periodicity is observed in each of the electromyography (EMG) signal (410) and the skin conductivity (GSR) signal (420), and that the baseline rhythm of the TDJ biosignal is measured stably. In the area marked in yellow, it can be seen that the directionality of the electromyography (EMG) signal (410) and the skin conductivity (GSR) signal (420) rising or falling simultaneously is synchronized.
[0114] In particular, it can be seen that the result (430) of the Cross-GAF conversion performed on the electromyogram (EMG) signal (410) and the skin conductivity (GSR) signal (420) has a significantly larger yellow area compared to the result (330) of the Cross-GAF conversion performed on the electrocardiogram (ECG) signal (310) and the brainwave (EEG) signal (320). This means that there are many parts where the directionality of the electromyogram (EMG) signal (410) and the skin conductivity (GSR) signal (420) is simultaneously synchronized, that is, it can be seen that there is a high correlation.
[0115] Meanwhile, the step (S130) of allocating computations by differentiating memory layers within the NPU (Neural Processing Unit) according to the importance of each of the multiple biosignal characteristics is a step of allocating memory layers within the NPU according to the importance of the biosignal characteristics, wherein biosignal data having important characteristics are placed in fast on-chip memory and biosignal data having additional characteristics are placed in off-chip memory to improve inference performance.
[0116] Specifically, the step (S130) of allocating computations by differentiating the memory hierarchy within the NPU (Neural Processing Unit) according to the importance of each of the multiple biosignal characteristics can optimize the memory within the NPU by analyzing the information entropy of the multiple biosignals and allocating memory.
[0117] FIG. 6 is a diagram illustrating a biosignal information entropy-based memory allocation process according to one embodiment of the present invention.
[0118] Referring to FIG. 6, when a biosignal segment is input, an entropy score for the segment can be calculated. Here, the entropy score of the biosignal segment can be calculated through the following Equation 12.
[0119]
[0120] Here, is a signal segment It refers to the information entropy of, is the signal value It refers to the normalized probability of occurrence.
[0121] These entropy scores quantify the uncertainty and complexity of information within biosignals; high entropy values indicate that the biosignal contains a wide range of values and thus a large amount of information, while low entropy values suggest that the biosignal is relatively simple and predictable.
[0122] For example, an ECG signal indicating a heart attack may have a higher entropy value than a normal heart rate pattern.
[0123] In addition, segments are classified based on diagnostic value according to these entropy scores, and memory hierarchies can be sequentially selected corresponding to the entropy values of the classified signal segments.
[0124] For example, if the entropy threshold is set to (4.5, 2.5), biosignals with an entropy score greater than 4.5 are determined to have high diagnostic value and may be preferentially allocated to the L1 cache memory that provides the fastest access speed in the NPU.
[0125] In addition, biosignals with an entropy score greater than 2.5 and less than or equal to 4.5 are determined to have an intermediate diagnostic value and may be allocated to the L2 cache memory, which is relatively slower than the L1 cache memory in the NPU.
[0126] In addition, biosignals with an entropy score of 2.5 or less are determined to have low diagnostic value and may be allocated to the slowest off-chip memory in the NPU.
[0127] FIG. 7 is a diagram illustrating an example of memory allocation of ECG signals by entropy according to one embodiment of the present invention.
[0128] Referring to FIG. 7, an ECG signal (510) having a regular pattern is shown, and its entropy score is 2.4, which is lower than 2.5 of the entropy thresholds (4.5, 2.5), so it can be determined that its diagnostic value is low and accordingly it is allocated to off-chip memory (511).
[0129] Additionally, it can be confirmed that the irregular ECG signal (520) with high variability has an entropy score of 2.62, which is greater than 2.5 and lower than 4.5 for the entropy threshold (4.5, 2.5), and thus has an intermediate diagnostic value, and is allocated to L2 cache memory (521).
[0130] Additionally, a normal ECG signal (530) with moderate variability can be determined to have a high diagnostic value as its entropy score is 4.6, which is greater than 4.5 for the entropy threshold (4.5, 2.5), and thus can be confirmed to be allocated to L1 cache memory (531).
[0131] Meanwhile, the step (S130) of allocating computations by differentiating the memory hierarchy within the NPU (Neural Processing Unit) according to the importance of each of the multiple biosignal characteristics can optimize the memory within the NPU by performing memory prefetching based on the periodic characteristics of each of the multiple biosignals.
[0132] Specifically, predictive memory prefetching is implemented by analyzing the periodic characteristics of biosignals and biological rhythms, and can be expressed by the following mathematical formula 13.
[0133]
[0134] Here, P(t+Δt) represents the data prediction required after time Δt, R(t) represents the circadian rhythm pattern, V(t) represents the variability of biological activity, and T(t) represents the temporal context element.
[0135] Furthermore, the circadian rhythm function R(t) is a deterministic function based on the periodic patterns of the human body, such as circadian rhythm, sleep-wake cycles, and post-meal states, and patterns can be captured through Fourier-based periodic analysis or simple statistical modeling.
[0136] Specifically, it is possible to analyze the periodic patterns of multiple biosignals, extract key circadian rhythm elements such as heart rate variability, breathing patterns, and activity levels, apply a rule-based prediction model in the time-frequency domain, and preemptively allocate memory based on predicted data access patterns.
[0137] Meanwhile, the step (S130) of allocating computations by differentiating the memory hierarchy within the NPU (Neural Processing Unit) according to the importance of each of the multiple biosignal characteristics can optimize the memory within the NPU by allocating memory based on the diagnostic priority of the multiple biosignals.
[0138] Specifically, as a memory management that considers the diagnostic importance and clinical priority of biosignals, it can be expressed by the following mathematical formula 14.
[0139]
[0140] Here, is a signal segment It refers to the diagnostic priority score, represents clinical importance, and implies diagnostic uniqueness, and represents the anomaly detection score, and represents the optimized weight for each type of biosignal.
[0141] And memory management based on the diagnostic priority of the signal segments calculated in this way can proceed as follows.
[0142] FIG. 8 is a diagram illustrating multilayer memory management based on biosignal diagnosis priority according to one embodiment of the present invention.
[0143] Referring to FIG. 8, a biosignal is acquired, the biosignal is segmented to calculate a diagnostic priority score for the signal segment, an abnormal sign score is detected, a priority is calculated, a memory hierarchy is reallocated based on the calculated priority, and the allocation order can be adjusted according to clinical urgency.
[0144] Meanwhile, the step (S130) of allocating computations by differentiating the memory hierarchy within the NPU (Neural Processing Unit) according to the importance of each of the multiple biosignal characteristics can optimize the memory within the NPU by allocating memory considering the correlation between the multiple biosignals.
[0145] Specifically, an integrated memory structure can be designed by utilizing the complementary characteristics of various biosignal modalities, and can be expressed by the following mathematical formula 15.
[0146]
[0147] Here, refers to an integrated memory structure, and refers to a specific biosignal modality, and represents the memory allocation weight of the corresponding modality, and represents the reusability indicator of the corresponding modality.
[0148] Considering the correlations between the modalities of these biosignals, the memory allocation method can be implemented as follows. For example, considering the correlation between the modalities of ECG and PPG signals, it can be managed as a cardiac cycle-based shared buffer from the perspective of cardiovascular synchronization; considering the correlation between the modalities of EEG and EMG signals, it can be managed as a cross-access cache from the perspective of neuromuscular interaction; and considering the correlation between the modalities of GSR and HRV signals, it can be managed as an event trigger memory pool from the perspective of autonomic nervous system response.
[0149] Meanwhile, the step (S140) of performing quantization corresponding to each of the multiple biosignal characteristics can maximize inference speed and power efficiency by applying a customized quantization technique that considers the characteristics of the biosignal to reduce the model size while maintaining accuracy.
[0150] Specifically, the step (S140) of performing quantization corresponding to each of the multiple biosignal characteristics can perform quantization by selecting a quantization bit width based on the clinical importance and frequency characteristics of each of the multiple biosignals.
[0151] FIG. 9 is a diagram illustrating a biosignal specialized quantization technique according to one embodiment of the present invention.
[0152] Referring to Fig. 9, the process of applying a differentiated quantization technique considering the clinical importance and frequency characteristics of the biosignal is explained.
[0153] Specifically, when model and correction data are input, signal sensitivity can be analyzed, optimal bits can be assigned per layer, and quantization corresponding to the assigned optimal bits can be applied.
[0154] For example, in the case of an ECG (QRS complex) signal, subtle changes in the shape of the QRS complex are important for diagnosing heart disease, and considering that the main frequency band is 10-25 Hz, the quantization bit width can be allocated to 8-12 bits, and quantization can be applied according to the allocated 8-12 bits.
[0155] In the case of EEG (alpha wave) signals, changes in alpha wave amplitude are necessary for cognitive status and diagnosis of central nervous system diseases, and considering that the main frequency band is 8-13 Hz, the quantization bit width can be allocated to 8-10 bits, and quantization can be applied according to the allocated 8-10 bits.
[0156] For EMG (activity indicator) signals, moderate precision is sufficient for measuring the degree of muscle activation, and considering that the main frequency band is 50-150 Hz, the quantization bit width can be allocated to 6-8 bits, and quantization can be applied according to the allocated 6-8 bits.
[0157] General background noise is background noise with low diagnostic value and can be represented with low precision. Considering that the main frequency band is also variable, the quantization bit width can be allocated to 4-6 bits, and quantization can be applied according to the allocated 4-6 bits.
[0158] The step (S140) of performing quantization corresponding to each of the multiple biosignal characteristics can perform quantization by selecting a bit width corresponding to the sensitivity of each of the multiple biosignals.
[0159] Specifically, the optimal bit width can be determined by analyzing the effect of each layer and weight of the neurotransmitter model on the characteristics of a specific biosignal, and the effect of the layer on the characteristics of a specific biosignal can be calculated through the following mathematical formula 16.
[0160]
[0161] Here, is a layer biosignal characteristics It signifies the importance of, represents the calculated importance score.
[0162] The higher the importance score calculated in this way, the higher the sensitivity, and the lower the importance score, the lower the sensitivity. When the importance is high (S > 0.8), the original precision is maintained (16 / 32 bits), when the importance is medium (0.3 < S ≤ 0.8), 8-bit quantization is applied, and when the importance is low (S ≤ 0.3), 4-bit or less quantization can be applied.
[0163] The step (S140) of performing quantization corresponding to each of the multiple biosignal characteristics can perform quantization by selecting a bit width corresponding to a specific frequency band of each of the multiple biosignals.
[0164] Specifically, it refers to a method in which filters and weights related to the frequency band of the biosignal maintain high precision, while other regions are quantized at low precision.
[0165] FIG. 10 is a diagram illustrating precision-preserving quantization by frequency band according to one embodiment of the present invention.
[0166] Referring to Fig. 10, when a model and signal specifications are input, a model layer is selected, and if it is a convolution layer, frequency band extraction and selection are performed. If the importance is higher than a threshold, precision is maintained, and if the importance is lower than a threshold, active quantization is applied to complete processing in the model layer.
[0167] For example, in the case of EEG signals, different bit widths are assigned to each frequency band for alpha (8-13Hz), beta (13-30Hz), and theta (4-8Hz) because each frequency characteristic has diagnostic importance; in the case of ECG signals, different bit widths are assigned to the frequency eodudrquf by applying differentiated precision to the QRS complex (10-25Hz) and P / T wave (0.5-10Hz) because each frequency characteristic has diagnostic importance; and in the case of PPG signals, different quantization can be applied to the basic pulse wave (0.5-5Hz) and 2nd / 3rd waveform (5-10Hz) because each frequency characteristic has diagnostic importance.
[0168] Meanwhile, although the aforementioned Cross-GAF conversion, NPU memory optimization, and biosignal specialized quantization were described as being performed on an NPU, they can also be applied to FPGAs, ASICs, etc., in addition to NPUs.
[0169] Meanwhile, the step (S120) of performing time series data preprocessing and Cross-GAF transformation according to one embodiment of the present invention may generate an enhanced Crpss-GAF image by combining information in the frequency domain and the time domain to minimize information loss occurring when visualizing time series data, and a Wavelet transform + GAF composite representation or Edge-preserving normalization may be applied.
[0170] In addition, the multi-biometric signal-based on-device AI inference optimization method according to one embodiment of the present invention can predict and limit power consumption in real time based on the type and complexity of the biometric signal and the prediction of NPU computation per inference cycle.
[0171] For example, power prediction can be performed based on the type, complexity, and NPU computation prediction of the biosignal per inference cycle through a learned regression model or LSTM, and high-precision inference can be performed only when the power consumed in inference is predicted to be below a certain level.
[0172] Meanwhile, a step (S150) of analyzing multiple biosignals through an optimized neural network model via memory hierarchy differentiation and quantization and outputting an inference result can be performed.
[0173] In addition, the above-described multi-biometric signal-based on-device AI inference optimization method may be provided as a computer-readable recording medium that records a program for executing the method on a computer.
[0174] FIG. 11 is a block diagram illustrating the configuration of a multi-biometric signal-based on-device AI inference optimization device according to one embodiment of the present invention.
[0175] Referring to FIG. 11, a multi-biometric signal-based on-device AI inference optimization device (100) according to one embodiment of the present invention may include a sensor unit (110) for collecting multi-biometric signals and a processor (120) that performs time-series data preprocessing and Cross-GAF conversion on the collected multi-biometric signals, assigns computations by differentiating memory layers within the NPU according to the importance of each multi-biometric signal characteristic, performs quantization corresponding to each multi-biometric signal characteristic, and analyzes the multi-biometric signals through an optimized neural network model through memory measurement differentiation and quantization to output an inference result.
[0176] Such a multi-biometric signal-based on-device AI inference optimization device (100) can be implemented as an on-device AI chipset, etc., attached to a desktop, server, or separate device.
[0177] Here, the multiple biosignals may include at least one of an electrocardiogram signal, a ballistocardiogram signal, an electroencephalogram signal, an electromyogram signal, a skin conduction signal, a photoplethysmogram signal, and a phonocardiogram signal.
[0178] And, the processor (120) can analyze the power spectral density of each of the multiple biosignals, perform filtering corresponding to each of the characteristics of the multiple biosignals, perform normalization corresponding to each of the amplitude characteristics of the multiple biosignals, perform synchronization between the multiple biosignals, and perform quality optimization of the multiple biosignals.
[0179] Additionally, the processor (120) can convert the interaction between multiple biosignals into a 2D image.
[0180] Additionally, the processor (120) can optimize memory within the NPU through at least one of the following methods: a method of allocating memory by analyzing the information entropy of multiple biosignals, a method of performing memory prefetching based on the periodic characteristics of each of the multiple biosignals, a method of allocating memory based on the diagnostic priority of the multiple biosignals, and a method of allocating memory by considering the correlation between the multiple biosignals.
[0181] Additionally, all of the above-described multi-biometric-based on-device AI inference optimization methods can be processed by a processor (120).
[0182] FIG. 12 is a block diagram illustrating the specific configuration of the multi-biometric signal-based on-device AI inference optimization device illustrated in FIG. 11.
[0183] Referring to FIG. 12, a multi-biometric signal-based on-device AI inference optimization device (100) may include a sensor unit (110), a processor (120), and a storage unit (130).
[0184] The processor (120) controls the overall operation of the multi-biometric-based on-device AI inference optimization device (100).
[0185] Specifically, the processor (120) includes RAM (121), ROM (122), main CPU (123), graphics processing unit (124), first to n interfaces (125-1 to 125-n), and a bus (126).
[0186] RAM (121), ROM (122), main CPU (123), graphics processing unit (124), first to n interfaces (125-1 to 125-n), etc. can be connected to each other via a bus (126).
[0187] The first to n interfaces (125-1 to 125-n) are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.
[0188] The main CPU (123) accesses the storage unit (130) and performs booting using the O / S stored in the storage unit (130). Then, it performs various operations using various programs, content, data, etc. stored in the storage unit (130).
[0189] In particular, the main CPU (123) can perform time-series data preprocessing and Cross-GAF conversion on the collected multiple biosignals, differentiate memory layers within the NPU according to the importance of each of the multiple biosignal characteristics to allocate operations, perform quantization corresponding to each of the multiple biosignal characteristics, and analyze the multiple biosignals through an optimized neural network model through memory measurement differentiation and quantization to output inference results.
[0190] A set of instructions for booting the system is stored in the ROM (122). When a turn-on command is input and power is supplied, the main CPU (123) copies the O / S stored in the storage unit (130) to the RAM (121) according to the instructions stored in the ROM (122), and executes the O / S to boot the system. When booting is complete, the main CPU (123) copies various application programs stored in the storage unit (130) to the RAM (121), and executes the application programs copied to the RAM (121) to perform various operations.
[0191] The graphics processing unit (124) generates a screen containing various objects such as icons, images, and text using a calculation unit (not shown) and a rendering unit (not shown). The calculation unit (not shown) calculates attribute values such as coordinate values, shape, size, and color for each object to be displayed according to the layout of the screen based on a received control command. The rendering unit (not shown) generates a screen of various layouts containing objects based on the attribute values calculated by the calculation unit (not shown).
[0192] In particular, the graphics processing unit (124) can implement objects generated by the main CPU (123) into a GUI (Graphic User Interface), icon, user interface screen, etc.
[0193] Meanwhile, the operation of the above-described processor (120) can be performed by a program stored in the storage unit (130).
[0194] The storage unit (130) stores various data, such as an O / S (Operating System) software module and various multimedia content, for operating a multi-biometric signal-based on-device AI inference optimization device (100).
[0195] In particular, the storage unit (130) may include a software module for performing time-series data preprocessing and Cross-GAF conversion on the collected multiple biosignals, differentiating memory layers within the NPU according to the importance of each of the multiple biosignal characteristics to allocate operations, performing quantization corresponding to each of the multiple biosignal characteristics, and analyzing the multiple biosignals through an optimized neural network model through memory measurement differentiation and quantization to output an inference result.
[0196] FIG. 13 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention.
[0197] Referring to FIG. 13, the storage unit (130) may store programs such as a time series data module (131), a Cross-GAF conversion module (132), a memory hierarchy differentiation module (133) within the NPU, and a quantization module (134).
[0198] The time series data module (131) can analyze the power spectral density of each of the multiple biosignals to perform filtering corresponding to each of the multiple biosignal characteristics, perform normalization corresponding to each of the multiple biosignals' amplitude characteristics, perform synchronization between the multiple biosignals, and optimize the quality of the multiple biosignals.
[0199] Additionally, the Cross-GAF conversion module (132) can convert the interaction between multiple biosignals into a 2D image form.
[0200] Additionally, the memory hierarchy differentiation module (133) within the NPU can optimize memory within the NPU through at least one of the following methods: a method of allocating memory by analyzing the information entropy of multiple biosignals, a method of performing memory prefetching based on the periodic characteristics of each of the multiple biosignals, a method of allocating memory based on the diagnostic priority of the multiple biosignals, and a method of allocating memory by considering the correlation between the multiple biosignals.
[0201] Additionally, the quantization module (134) can perform quantization through at least one of the following methods: selecting a quantization bit width based on the clinical importance and frequency characteristics of each of the multiple biosignals, selecting a bit width corresponding to the sensitivity of each of the multiple biosignals, and selecting a bit width corresponding to a specific frequency band of each of the multiple biosignals.
[0202] Meanwhile, a non-transitory computer-readable medium storing a program that sequentially performs the control method according to the present invention may be provided.
[0203] A non-transient readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on non-transient readable media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.
[0204] In addition, although the bus is not shown in the block diagram described above illustrating a multi-biometric signal-based on-device AI inference optimization device, a processor such as a CPU or microprocessor may be further included to perform time-series data preprocessing and Cross-GAF transformation on the collected multi-biometric signals, differentiate memory layers within the NPU according to the importance of each multi-biometric signal characteristic to allocate computations, perform quantization corresponding to each multi-biometric signal characteristic, and analyze the multi-biometric signals through an optimized neural network model via memory measurement differentiation and quantization to output inference results.
[0205] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.
Claims
1. In a multi-biometric signal-based on-device AI inference optimization method, Step of collecting multiple biosignals; A step of performing time-series data preprocessing and Cross-GAF conversion on the above multiple biosignals; A step of allocating computations by differentiating memory hierarchy within the NPU (Neural Processing Unit) according to the importance of each of the aforementioned multiple biosignal characteristics; A step of performing quantization corresponding to each of the above multiple biosignal characteristics; and A method for optimizing on-device AI inference based on multiple biosignals, comprising the step of analyzing multiple biosignals through a neural network model optimized through memory layer differentiation and quantization and outputting an inference result.
2. In Paragraph 1, A method for optimizing on-device AI inference based on multiple biosignals, wherein the multiple biosignals include at least one of an electrocardiogram signal, a ballistocardiogram signal, a brainwave signal, an electromyogram signal, a skin conduction signal, a photoplethysmogram signal, and a phonocardiogram signal.
3. In Paragraph 2, The step of performing the above time series data preprocessing and Cross-GAF transformation is: A multi-biometric signal-based on-device AI inference optimization method comprising analyzing the power spectral density of each of the multi-biometric signals, performing filtering corresponding to each of the characteristics of the multi-biometric signals, performing normalization corresponding to the amplitude characteristics of each of the multi-biometric signals, performing synchronization between the multi-biometric signals, and performing quality optimization of the multi-biometric signals.
4. In Paragraph 3, The step of performing the above time series data preprocessing and Cross-GAF transformation is: A multi-biometric signal-based on-device AI inference optimization method that converts the interaction between the above multi-biometric signals into a 2D image form.
5. In Paragraph 4, The step of allocating computations by differentiating memory hierarchies within the NPU according to the importance of each of the aforementioned multiple biosignal characteristics is: A method for optimizing memory within an NPU based on multiple biosignals, wherein memory is optimized through at least one of the following: a method of allocating memory by analyzing the information entropy of the multiple biosignals, a method of performing memory prefetching based on the periodic characteristics of each of the multiple biosignals, a method of allocating memory based on the diagnostic priority of the multiple biosignals, and a method of allocating memory by considering the correlation between the multiple biosignals.
6. In Paragraph 5, The step of performing quantization corresponding to each of the above multiple biosignal characteristics is: A method for optimizing on-device AI inference based on multiple biosignals, wherein quantization is performed through at least one of the following: a method for selecting a quantization bit width based on the clinical importance and frequency characteristics of each of the multiple biosignals, a method for selecting a bit width corresponding to the sensitivity of each of the multiple biosignals, and a method for selecting a bit width corresponding to a specific frequency band of each of the multiple biosignals.
7. A computer-readable recording medium storing a program for executing on a computer a multi-biometric signal-based on-device AI inference optimization method described in any one of claims 1 to 6.
8. In a multi-biometric signal-based on-device AI inference optimization device, A sensor unit for collecting multiple biosignals; and A multi-biometric signal-based on-device AI inference optimization device comprising: a processor that performs time-series data preprocessing and Cross-GAF transformation on the collected multi-biometric signals, differentiates memory hierarchy within the NPU according to the importance of each of the multi-biometric signal characteristics to allocate computations, performs quantization corresponding to each of the multi-biometric signal characteristics, and analyzes the multi-biometric signals through a neural network model optimized through the memory measurement differentiation and quantization to output an inference result.
9. In Paragraph 8, A multi-biometric signal-based on-device AI inference optimization device wherein the multi-biometric signals include at least one of an electrocardiogram signal, a ballistocardiogram signal, a brainwave signal, an electromyogram signal, a skin conduction signal, a photoplethysmogram signal, and a phonocardiogram signal.
10. In Paragraph 9, The above processor is, A multi-biometric signal-based on-device AI inference optimization device that analyzes the power spectral density of each of the multi-biometric signals, performs filtering corresponding to each of the characteristics of the multi-biometric signals, performs normalization corresponding to the amplitude characteristics of each of the multi-biometric signals, performs synchronization between the multi-biometric signals, and performs quality optimization of the multi-biometric signals.
11. In Paragraph 10, The above processor is, A multi-biometric signal-based on-device AI inference optimization device that converts the interaction between the above multi-biometric signals into a 2D image form.
12. In Paragraph 11, The above processor is, A multi-biometric signal-based on-device AI inference optimization device that optimizes memory within an NPU through at least one of the following methods: a method of allocating memory by analyzing the information entropy of the multi-biometric signals, a method of performing memory prefetching based on the periodic characteristics of each of the multi-biometric signals, a method of allocating memory based on the diagnostic priority of the multi-biometric signals, and a method of allocating memory by considering the correlation between the multi-biometric signals.
