Hierarchical architecture-based real-time analysis method for multi-modal wearable sensor data
Patent Information
- Application Number
- PCT/CN2025/129127
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-03-14
- Filing Date
- 2025-10-22
- Publication Date
- 2026-09-17
Smart Images
Figure CN2025129127_17092026_PF_FP_ABST
Abstract
Description
A Real-Time Analysis Method for Multimodal Wearable Sensor Data Based on Hierarchical Architecture Technical Field
[0001] This invention relates to the field of sensor signal processing technology, and in particular to a method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture. Background Technology
[0002] In assessing the athletic performance of healthy individuals and the rehabilitation of patients with chronic diseases, the cardiopulmonary and circulatory systems play a crucial role in the response during exercise. With the rise in cardiovascular disease, diabetes, obesity, and other chronic illnesses, the stress on the cardiopulmonary system is also increasing. Real-time monitoring can more accurately assess individual endurance and metabolic efficiency, which is not only beneficial for athletes in designing personalized training programs but also of great significance for health management in the general population and the rehabilitation of patients with chronic diseases.
[0003] The selection of sensors and the integration of multi-sensor data are crucial because wearable systems generate a large amount of diverse data from devices such as heart rate monitors, respiration sensors, and accelerometers. Effective data fusion is essential for comprehensive analysis. Currently available sensor monitoring devices generally suffer from the following problems: most require specialized medical background knowledge for operation and maintenance, making them unsuitable for everyday use by ordinary users; traditional sensor monitoring devices are large and heavy, making them difficult to carry and hindering all-day monitoring; wearing these devices may disrupt users' daily lives and work, reducing user compliance; and professional medical devices are expensive, making them unaffordable for ordinary users.
[0004] Existing sensor monitoring systems have revealed many shortcomings in addressing the aforementioned challenges, particularly in smart city construction and large-scale data processing. Traditional data transmission methods are unable to handle the efficient transmission of massive amounts of real-time user data, often resulting in data loss or delays, severely impacting system reliability. Simultaneously, the lack of specialized algorithms and tools for big data analysis makes extracting valuable information from massive datasets difficult. Furthermore, existing systems have limited scalability, making it difficult to adapt to the rapidly growing user demands in smart cities. In such complex environments as smart cities, the introduction of big data processing and transmission frameworks is particularly crucial.
[0005] Inaccurate data can compromise the reliability of cardiopulmonary function analyses, necessitating a robust distributed storage architecture to ensure data integrity and protection throughout the collection, transmission, and storage process, especially in large-scale data environments. To address these challenges, data processing efficiency can be optimized by integrating edge, fog, and cloud computing systems, leveraging efficient computing and transmission capabilities, and scalable storage architectures. This approach enables rapid analysis of the vast amounts of data generated by wearable devices, supporting subsequent analysis and informed decision-making.
[0006] In view of this, the present invention proposes a real-time analysis method for multimodal wearable sensor data based on a hierarchical architecture to solve the above problems and meet the needs of daily sensor monitoring and multi-user data transmission and analysis in the context of smart cities. Summary of the Invention
[0007] To address the aforementioned problems, this invention provides a method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture.
[0008] In a first aspect, the present invention provides a real-time analysis method for multimodal wearable sensor data based on a hierarchical architecture, which adopts the following technical solution:
[0009] A method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture includes:
[0010] Acquire sensor data, including inertial navigation unit data acquisition, goniometer data acquisition, and electrocardiogram data acquisition.
[0011] Synchronize the acquired sensor data signals;
[0012] The synchronized data signals are transmitted in layers, with sensor data processing performed sequentially at the edge layer and fog layer before being transmitted to the cloud layer; sensor data modeling and comprehensive analysis are then performed in the cloud layer.
[0013] Store the results of data analysis and processing.
[0014] Furthermore, the data signal synchronization of the acquired sensor data includes data synchronization for multiple sensors of the same type during transmission, including data transmission synchronization of multiple inertial navigation units and data transmission synchronization of multiple goniometers. For the inertial navigation unit, the receiving time is captured by an event timer, the propagation delay is calculated, and bidirectional delay measurement is used to eliminate channel asymmetry error. The integrity of the synchronized data is verified by fusion based on a Kalman filter. For the goniometer, the linear relationship between the master and slave clocks is fitted by the least squares method, and nonlinear correction is performed based on clock deviation compensation. The compensation formula is: a_corrected=a·[1+γ(T-T0)], where γ is the crystal temperature coefficient and T0 is the reference temperature.
[0015] Furthermore, the sensor data processing is performed sequentially at the edge layer and the fog layer, including inertial navigation unit (INS) data processing at the edge layer. Specifically, the INS data is denoised using a Kalman filter algorithm. For each dimension of the signal, the Kalman filter algorithm is used for denoising. The system is then predicted for the next time step, and the state covariance is predicted based on the state transition matrix. Finally, the Kalman gain K is calculated based on the predicted covariance and the observation noise covariance. k Used to weigh predicted and observed values.
[0016] Using the calculated Kalman gain K k Combined with the observed value Z k Update the predicted state to obtain the optimal estimated state at time k. At the same time, update the optimal estimated covariance.
[0017] Furthermore, the sequential processing of sensor data in the edge layer and fog layer also includes processing goniometer data in the edge layer. Specifically, the square root of the two channels of each data segment of a single goniometer is calculated, the signals of the two channels are merged, the continuous respiratory signal data is divided into multiple data segments according to time periods, a composite respiratory signal is generated by an energy fusion algorithm for multiple goniometer data streams, and the dynamic weight SNR of each goniometer signal within the sliding window is calculated.
[0018] Furthermore, the sequential processing of sensor data in the edge layer and fog layer also includes processing electrocardiogram (ECG) data in the edge layer. Specifically, baseline drift correction is performed on the acquired ECG data, a high-pass Butterworth filter is used to separate baseline drift below 0.5 Hz, and then the QRS complex is enhanced by using a bandpass filter to specifically enhance the QRS wave energy while suppressing 20-500 Hz electromyographic noise and low-frequency T-wave components below 5 Hz.
[0019] Furthermore, the sensor data processing performed sequentially at the edge layer and the fog layer includes inertial navigation unit data processing in the fog layer. Specifically, the acceleration data of the inertial navigation unit is processed to mitigate the influence of gravity on motion classification, as follows:
[0020] Where H(s) represents the transfer function, s represents the complex frequency variable in the Laplace transform, and Q... k The quality factor is ω0, and the center angular frequency is ω0. These are filtered triaxial acceleration data. It is the acceleration data after removing the influence of gravity.
[0021] Furthermore, the sequential sensor data processing at the edge layer and fog layer also includes goniometer data processing at the fog layer. This involves using a hybrid filtering architecture to denoise the fused goniometer data, dynamically adjusting the time window length, prioritizing the retention of low-frequency components of the respiratory signal, and employing a Butterworth filter with a cutoff frequency of 10Hz to eliminate high-frequency interference from muscle tremors. After denoising, the data undergoes baseline correction and normalization, morphological baseline extraction is used, and opening and closing operations are performed using structuring elements matched to the respiratory cycle to separate slowly changing baseline drift. Respiratory dynamics analysis is then performed to reveal the hidden dynamic characteristics of respiration through the respiratory signal. The maximum Lyapunov exponent is calculated to measure the sensitivity of the respiratory cycle to initial conditions; a higher value indicates a more unstable rhythm.
[0022] Furthermore, the sequential sensor data processing at the edge layer and fog layer also includes ECG data processing at the fog layer. Specifically, QRS peaks in the ECG data are detected by introducing a dynamic integration window in the Pan-Tompkins algorithm, allowing the window length to automatically adjust with heart rate and avoiding waveform overlap at high heart rates. Then, after performing squaring and moving integration, time-frequency domain features are extracted from the ECG signal to calculate the continuous RR interval sequence {Δt}. i The power spectral density was estimated using the Lomb-Scargle algorithm, and finally, respiratory-electrocardiogram coupling analysis was performed to establish a phase synchronization model between the respiratory signal D_norm(t) and the RR interval.
[0023] Furthermore, the sensor data modeling and comprehensive analysis in the cloud includes inertial navigation unit (INS) data modeling in the cloud. The INS data modeling includes modeling using a DCNN classifier, employing six progressively increasing modules. The first three modules use a residual Inception structure, while the last three modules introduce dynamically deformable convolutions to capture irregular motion features. The number of output channels for each module gradually increases from 64 to 512. An SE attention mechanism is added after the convolution to enhance the response of key features. A cross-modal interaction layer is inserted after the third and fifth modules to fuse the spatiotemporal correlation between acceleration and angular velocity through a cross-attention mechanism.
[0024] Furthermore, the aforementioned sensor data modeling and comprehensive analysis in the cloud layer also includes goniometer data modeling in the cloud layer. This involves using a distributed computing framework to perform Fourier transform, converting the signal from the time domain to the frequency domain, and using clustering algorithms to automatically identify the main frequency components in the frequency spectrum to determine the respiratory rate; calculating the signal amplitude spectrum and storing intermediate results using a cloud-based distributed storage system; calculating the respiratory cycle (the time interval between two adjacent breaths) by detecting the peaks or troughs of the respiratory waveform; and decomposing the respiratory signal into wavelet coefficients of different scales to analyze the waveform characteristics of the signal, and then performing comprehensive data analysis and long-term storage.
[0025] In summary, the present invention has the following beneficial technical effects:
[0026] This invention collects human body signals, including motion state detection and classification, respiratory state analysis, and electrocardiogram (ECG) signal monitoring. These different models and data processing methods are allocated to different levels according to their processing difficulty and computational requirements, ensuring the overall efficiency of the model and algorithm. Through this method, real-time monitoring and analysis of user sensor signals can be achieved, providing users with detailed health status reports and supporting data management and analysis needs in multi-user environments, thus promoting the development of smart city health management. Attached Figure Description
[0027] Figure 1 is a schematic diagram of a real-time analysis method for multimodal wearable sensor data based on a hierarchical architecture according to Embodiment 1 of the present invention;
[0028] Figure 2 is a schematic diagram of the overall Apache-Flink structure of Embodiment 1 of the present invention. Detailed Implementation
[0029] The present invention will be further described in detail below with reference to the accompanying drawings.
[0030] Example 1
[0031] Referring to Figure 1, this embodiment of a method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture includes:
[0032] Acquire sensor data, including inertial navigation unit data acquisition, goniometer data acquisition, and electrocardiogram data acquisition.
[0033] Synchronize the acquired sensor data signals;
[0034] The synchronized data signals are transmitted in layers, with sensor data processing performed sequentially at the edge layer and fog layer before being transmitted to the cloud layer; sensor data modeling and comprehensive analysis are then performed in the cloud layer.
[0035] Store the results of data analysis and processing.
[0036] Specifically:
[0037] This embodiment describes the multimodal data hierarchical processing flow and computational modeling process. It details the processing flow, modeling process, and result generation steps for acquiring multimodal signals from five inertial navigation units, three goniometers, and an electrocardiogram sensor worn by the user, as shown in Figure 1.
[0038] S1. Data Acquisition
[0039] The user wears all the sensors on the corresponding locations on their body and begins data collection.
[0040] S1.1 Inertial Navigation Unit Data Acquisition
[0041] Each user wears five inertial navigation units (INS) located on the back, left upper arm, right upper arm, left thigh, and right thigh, comprehensively recording the physical information of all parts of the body during movement. The INS data is represented by X... α (t)∈R 45 It means that X α (t)=[X α1 (t),X α2 (t),X α3 (t),X α4 (t),X α5 [(t)], corresponding to 5 inertial navigation units, where X αi (t)∈R 9 (i = 1 to 5) are composed of three dimensions (R) 3 acceleration signals, three-dimensional (R) 3 ) gyroscope signals and three-dimensional (R 3 The signal is composed of magnetometer signals.
[0042] S1.2 Goniometer Data Acquisition
[0043] Each user wears three goniometers to record the angular changes of the chest cavity in two spatial planes. These angular change parameters are represented by θ1(t), θ2(t), and θ3(t), corresponding to the three different goniometers, where θ i (t)∈R 2 (i = 1, 2, 3). The time series for data acquisition is set as t0, t1, ..., tt N The sampling frequency is Where T s The sampling period is θ1(t) θ2(t) θ3(t) . The goniometer data matrix can be represented as: Θ(t)=[θ1(t) θ2(t) θ3(t)] T
[0044] S1.3 ECG Data Acquisition
[0045] Each user wears a single-lead ECG sensor with the electrode aligned with the V2 lead on the chest. The signal is represented by E(t), where E(t)∈R. 1 The ECG sensor has a fixed sampling frequency of 500Hz.
[0046] S2. Data transmission from sensor to edge layer
[0047] All sensor data is transmitted to the mobile device via Bluetooth. For multiple sensors of the same type, data synchronization is achieved during transmission, including data transmission synchronization of multiple inertial navigation units and multiple goniometers. The specific implementation is as follows:
[0048] The mobile device, acting as both the master device and the edge layer in the hierarchical transmission framework, provides a clock reference to multiple inertial navigation units (INS) via Bluetooth. Slave devices synchronize based on the master device's clock, ensuring the synchronization of timestamps across different INS units during data transmission. Specifically:
[0049] The central inertial navigation unit is designated as the master device, and the other units are designated as slave devices. The master device establishes a star topology network through the Bluetooth Low Energy 5.1 protocol to form a 1:M connection mode.
[0050] The master device utilizes the 28-bit CLK clock built into the Bluetooth baseband layer to embed the following in periodic broadcast packets: the master device's local clock value, the most recent clock correction timestamp, and the clock drift compensation coefficient. The slave device captures the reception time through an event timer, calculates the propagation delay, and uses bidirectional delay measurement to eliminate channel asymmetry errors.
[0051] A second-order clock model is established based on the fusion of clock phase difference observations, crystal oscillator frequency stability, and temperature drift characteristics using a Kalman filter.
[0052] Where θ(t) represents the cumulative phase deviation of the slave device clock relative to the master device reference clock at time t, θ0 represents the initial master-slave clock deviation measured by timestamp exchange at system startup, f0 represents the theoretical operating frequency of the inertial navigation unit crystal oscillator, α represents the frequency aging coefficient, and β represents the frequency drift acceleration coefficient.
[0053] An IEEE 1588 boundary clock check is performed every 100ms. CRC-16 is used to verify the integrity of the synchronized data, and a synchronization error threshold is set. If the threshold is exceeded, an emergency resynchronization is triggered.
[0054] The goniometer's data transmission synchronization is achieved by constructing a synchronization broadcast network through the extended broadcast channel of the mobile device. Each goniometer is dynamically allocated a dedicated time slot for transmitting angle data. The mobile device periodically sends a reference pulse signal as a hardware-level time reference. The goniometer obtains its master clock reference by listening to the broadcast packets and uses its hardware interrupt pin to capture the rising edge of the reference pulse, thus achieving initial synchronization.
[0055] Clock skew compensation: Every 50ms, the mobile device sends a synchronization packet (t1, t2, t3) containing three consecutive time reference points to all goniometers. The goniometers record the receiving time (t1', t2', t3') based on their local crystal oscillator and fit a linear relationship between the master and slave clocks using the least squares method: Δt = a·t + b, where a represents the frequency drift rate and b is the initial offset. This model performs nonlinear correction once per hour, and the compensation formula is: a_corrected = a·[1 + γ(T - T0)], where γ is the crystal oscillator temperature coefficient and T0 is the reference temperature.
[0056] When the goniometer sends angle data, it embeds the predicted clock deviation Δt into the data packet header. After the mobile device receives the data, it dynamically adjusts the time slot allocation based on the difference between the current global time and the timestamp in the packet header.
[0057] S3. Edge Layer Data Processing, Modeling, and Transmission
[0058] Mobile devices, acting as the edge layer of the hierarchical network model, perform preliminary processing and modeling of all sensor data and transmit the data based on the hierarchical transmission framework.
[0059] S3.1 Edge Layer Data Processing and Modeling
[0060] S3.11 Edge Layer Inertial Navigation Unit Data Processing and Modeling
[0061] First, the inertial navigation unit data is denoised. For each dimension of the signal, the Kalman filter algorithm is used for denoising.
[0062] To predict the next time step of the system, based on the state transition matrix:
[0063] in It is the optimal estimate from the previous moment, F k B is the system's state transition matrix. k It is the system's control input matrix, u k It is the system's control input.
[0064] Predicted state covariance P k|k-1 for:
[0065] Among them, Qk It is the process noise covariance matrix.
[0066] Calculate the Kalman gain K based on the prediction covariance and the observation noise covariance. k The formula used to balance predicted and observed values is:
[0067] Among them, H k Let R be the observation matrix. k To observe the noise covariance matrix.
[0068] Using the calculated Kalman gain K k Combined with the observed value Z k Update the predicted state to obtain the optimal estimated state at time k. The calculation formula is:
[0069] Simultaneously update the optimal estimated covariance P. k|k-1 P k|k =(IK k H k )P k|k-1
[0070] The above Kalman filter algorithm effectively suppresses noise in the data and obtains a more accurate state estimate.
[0071] L2-norm is used to quantize the amplitude of a signal, reflecting its overall strength. Using L2-norm theory, the L2 norm of the inertial navigation unit data after filtering is calculated, specifically for acceleration and angular velocity.
[0072] Next, the L2 norms of acceleration and angular velocity are weighted and summed to integrate acceleration and angular velocity information, thereby enhancing the robustness of motion classification. S=α·||a||2+β·||w||2
[0073] Where α and β are weighting coefficients. The threshold T is set by calculating the L2 norm statistic of the static data: T = μ static +kσ static
[0074] Where μ static Let σ be the mean of the L2 norm of the static data. static denoted as , where is the standard deviation of the L2 norm of static data, and k is the confidence coefficient. When the sum of the L2 norms S at the current moment is input, it is compared with a set threshold. If the sum is less than the threshold, the human body is judged to be in a static state; if the sum is greater than the threshold, the human body is judged to be in a dynamic state, thus classifying the human body into static and dynamic states.
[0075] S3.12 Edge Layer Goniometer Data Processing
[0076] Preliminary data processing is performed on the edge layer device. The square root of the two channels of each data segment from a single goniometer is calculated separately. The signals from the two channels are then merged. Continuous respiratory signal data is divided into multiple data segments according to time periods. Each segment contains signal variation values within a certain time range. Assume each data segment is D. i (t k ), angular change value θ i (t), where t k The k-th time segment is represented as:
[0077] A composite respiratory signal is generated from the data streams of the three goniometers using an energy fusion algorithm. The SNR of each goniometer signal within a sliding window is calculated, and the weights are dynamically allocated.
[0078] S3.13 Edge Layer ECG Data Processing
[0079] First, baseline drift correction was performed on the acquired ECG data using a second-order high-pass Butterworth filter. This filter, with its maximum flatness within the passband, can accurately separate baseline drift below 0.5 Hz (primarily caused by respiratory motion and electrode contact impedance changes) without affecting the QRS complex morphology. The transfer function is defined as:
[0080] Then, the QRS complex is enhanced using a 5-15Hz bandpass filter to specifically amplify the QRS wave energy (mainly distributed in the 10-25Hz range) while suppressing electromyographic noise (20-500Hz) and low-frequency T-wave components (<5Hz). The transfer function satisfies:
[0081] Finally, real-time heart rate estimation is performed using a dynamic threshold peak detection algorithm, which adaptively adjusts the detection threshold within a sliding window to overcome the influence of signal amplitude fluctuations caused by exercise.
[0082] S3.2 Edge Layer Data Transmission
[0083] The edge device categorizes the preprocessed data by topic, including: acceleration, angular velocity, goniometer data, and ECG data. Data is sent to a Kafka cluster. It uses Avro serialization format, with added timestamps and device ID metadata. Topics are hashed by device ID to ensure data ordering for the same user and support parallel Kafka consumption.
[0084] S4. Fog Data Processing, Modeling, and Transmission
[0085] To achieve more efficient data processing and more accurate behavioral analysis, local servers with moderate computing power were chosen as fog layer nodes. As the core layer connecting edge devices and the cloud, the fog layer is responsible for in-depth analysis of the input data. Compared to edge devices, the fog layer possesses more powerful computing capabilities, supporting complex signal processing algorithms and model training, thus enabling more detailed classification of human behavior. By establishing and optimizing behavioral classification models in the fog layer, reliance on cloud computing power can be effectively reduced, data transmission latency can be decreased, and strong support can be provided for application scenarios with high real-time requirements.
[0086] S4.1 Fog Data Reception
[0087] The Kafka consumer group in the fog computing layer consists of multiple fog computing nodes, responsible for consuming preprocessed data sent from the edge layer to Kafka in real time, including: acceleration, angular velocity, goniometer data, and ECG data. Kafka topics are hashed and partitioned by device ID (e.g., each user has their own partition) to ensure that data for the same user is processed by the same consumer, maintaining time-series consistency. The data consumption process begins with periodic poll() operations. Fog nodes pull data in batches at 100ms intervals. During the deserialization stage, the raw byte stream in Avro format is converted into structured objects (containing fields such as timestamp, device ID, and data value). Subsequently, the system dynamically adjusts the processing logic based on the real-time identified human activity type (including "dynamic" and "static"): for dynamic activities, a 5-second sliding window (1-second step) is used to continuously calculate the correlation coefficient between peak heart rate and cadence; for static activities, a 10-second rolling window is triggered to calculate the average respiratory rate.
[0088] S4.2 Fog Layer Inertial Navigation Unit Data Modeling
[0089] The acceleration data of the inertial navigation unit is processed to reduce the influence of gravity on motion classification. The specific processing method is as follows:
[0090] Where H(s) represents the transfer function, s represents the complex frequency variable in the Laplace transform, and Q... k The quality factor is ω0, and the center angular frequency is ω0. These are filtered triaxial acceleration data. It is the acceleration data after removing the influence of gravity.
[0091] The processed acceleration data is summed of squares, and the triaxial acceleration data is merged into a single acceleration data set. The continuous acceleration signal data is divided into multiple data segments according to time intervals, each segment containing signal change values within a certain time range. Let each data segment be G. i (t k ), where tk The k-th time segment is represented as:
[0092] Different static actions exhibit significant differences within the inertial navigation unit, resulting in a relatively low complexity for the classification model. Therefore, specific static behavior classification was chosen to be performed within the fog layer. Acceleration G i (t k Both angular velocity and angular velocity are inputs to the designed deep convolutional neural network (DCNN) classifier. The architecture consists of two convolutional modules, each containing convolution operations, batch normalization, and a ReLU activation function. The convolutional operations use a 5x5 window. Additionally, the architecture includes a culling layer, a fully connected layer, and a flexible maximum transfer function (WMP). The WMP classifies activities by calculating the probability associated with each input in the WMP. The activity with the highest probability is selected as the prediction. The learning rate decrease factor is set to 0.01.
[0093] The network consists of four convolutional layers with progressively increasing output channel numbers (18, 36, 72, and 100). Each convolutional layer is followed by a batch normalization layer and a max-pooling layer. The kernel sizes of the convolutional layers are (3,3), (2,3), (2,2), and (2,2), with corresponding spans and padding to preserve spatial information. After the convolutional layers, the feature maps are flattened and fed into two fully connected layers with 180 and 3 neurons, respectively. A culling layer with a probability of 0.2 is added before the first fully connected layer to prevent overfitting. The network is trained using an adaptive optimizer with a learning rate of 0.02, 20 iterations, and a batch size of 128. This architecture leverages the spatial and temporal dependencies of sensor data to effectively classify human static activities.
[0094] S4.3 Fog Layer Goniometer Data Processing
[0095] A hybrid filtering architecture is used to denoise the fused goniometer data, dynamically adjusting the time window length to prioritize the preservation of low-frequency components of the respiratory signal. The window length L(t) is adjusted based on the real-time calculated signal power spectral density (PSD). Data within the window is weighted according to Gaussian distribution.
[0096] A Butterworth filter with a cutoff frequency of 10Hz is used to eliminate high-frequency interference such as muscle tremors. Fast convolution operations are achieved through preset filter coefficients. Kalman optimization is performed simultaneously to establish a two-state model of respiratory motion (angle value + rate of change), and signal tracking capability is continuously optimized through a noise parameter adaptive mechanism. When sudden interference is detected, the observation noise weight is automatically reduced to improve prediction stability.
[0097] Equations of state:
[0098] Observation equation: z k =[1 0]x k +v k ,v k ~N(0,R)
[0099] Online updates of process noise covariance matrix and observation noise variance:
[0100] in Let λ be the variance of the second derivative of the signal, and λ = 0.95 be the forgetting factor.
[0101] After noise reduction, the data undergoes baseline correction and normalization. Morphological baseline extraction is employed, and opening and closing operations are performed using a structure element matched to the respiratory cycle to separate slowly changing baseline drift. A flat structure element g[m] with length L is defined. g =2×round(F s / f resp ), f resp This is an estimate of the current respiratory rate. D corrected =D filtered -D base
[0102] Dynamic amplitude normalization linearly maps the signal to the [-1,1] interval within a sliding window, preserving the relative waveform shape while eliminating sensor sensitivity differences.
[0103] Respiratory dynamics analysis was then performed to reveal the hidden dynamic characteristics of respiration through respiratory signals. The maximum Lyapunov exponent was calculated to measure the sensitivity of the respiratory cycle to initial conditions; a higher value indicates a more unstable rhythm. Rapid estimation was achieved by tracking the divergence rate of adjacent phase trajectories, avoiding complex matrix operations.
[0104] S4.4 Fog Layer ECG Data Processing
[0105] First, QRS complexes in the ECG data are detected using adaptive QRS detection. A dynamic integration window is introduced into the Pan-Tompkins algorithm, allowing the window length to automatically adjust with heart rate and avoid waveform overlap at high heart rates. The improved Pan-Tompkins algorithm consists of three steps. The first step involves differentiation to enhance the QRS slope.
[0106] The second step involves squaring the components to highlight the high-frequency parts. S[n] = W[n] 2
[0107] The third step is moving integration, which uses a dynamic integration window to further smooth the noise.
[0108] Next, time-frequency domain feature extraction is performed on the electrocardiogram signal, and the continuous RR interval sequence {Δt} is calculated. i The power spectral density P(f) is estimated using the Lomb-Scargle algorithm.
[0109] SDNN reflects the overall autonomic nervous system tension, and the Lomb-Scargle method overcomes the problem of non-uniform sampling during the RR interval and accurately quantifies the LF / HF power ratio.
[0110] Finally, respiratory-electrocardiogram coupling analysis was performed to establish respiratory signal D. norm (t) Phase synchronization model with RR interval:
[0111] Multimodal fusion was performed to quantify respiratory sinus arrhythmia (RSA) using the phase synchronization index ρ, revealing the vagus nerve-mediated cardiopulmonary coupling mechanism.
[0112] S4.5 fog data transmission
[0113] After real-time processing and context-aware aggregation are completed at the fog computing layer, the system transmits the data to the cloud computing layer via a layered Kafka topic strategy. Fog nodes, acting as producers, divide the processed data into two streams: acceleration and angular velocity data from the inertial navigation unit, respiratory signals processed from goniometer data, and processed ECG data. One type of publishing uses Avro serialization with added compression identifiers (LZ4 algorithm compression rate 60%). Each message includes the device ID, time window start and end stamps, and aggregated metrics in JSON format. The other type is raw data cold backup, targeting raw sensor waveforms (such as ECG fragments) requiring long-term storage. These are hashed and partitioned by user ID and written to the ` / batch / data` topic. The number of partitions dynamically matches the cloud storage cluster node size to avoid write hotspots. During transmission, FogLayer ensures messages are delivered only once using Kafka's idempotent producers, while SSL encryption and ACL access control are enabled to ensure the privacy and compliance of medical data.
[0114] S5 cloud data analysis, storage, and model building
[0115] The cloud layer plays a crucial role in the entire hierarchical algorithm. The central control system is chosen as the cloud layer because it has powerful computing capabilities and storage space, can efficiently process large-scale data, support complex model training and inference, can realize dynamic action classification, perform comprehensive analysis and long-term storage of other data, and meet the requirements of real-time performance, collaboration and scalability.
[0116] S5.1 Cloud Data Reception
[0117] The cloud computing layer pulls two types of data from the fog layer's topics via the Kafka Connect service: pre-aggregated summary data from the fog layer (including...) This system provides a long-term archive of raw sensor data. Summary data is directly stored in a time-series database for rapid retrieval, while raw data is distributed and written to an HDFS cluster for cold storage, with a retention period of up to 6 months to support backtesting analysis. For batch processing tasks, the system uses Apache Spark to build an offline pipeline, triggering global feature extraction hourly—such as cross-user cardiopulmonary function correlation analysis (e.g., the distribution relationship between heart rate recovery rate and age). Simultaneously, a personalized exercise prescription generation model is trained using the XGBoost algorithm. The trained model is packaged as a microservice in ONNX format, deployed to a Kubernetes cluster, and interacts with the front-end health management platform via a REST API.
[0118] S5.2 Cloud Inertial Navigation Unit Data Modeling
[0119] The dynamic activity classification task is executed in the cloud, leveraging the powerful computing capabilities and distributed storage of cloud computing. Furthermore, cloud training supports large-scale dataset processing, parallel model training, and real-time inference, meeting the high computing resource requirements of dynamic activity classification.
[0120] Building upon the classifier developed for static motion, a novel and more complex DCNN classifier is designed. The four basic convolutional modules are expanded to six progressively increasing modules. The first three modules employ a residual Inception structure (containing 1x1, 3x3, and 5x5 parallel convolutional kernels), while the latter three modules introduce dynamically deformable convolutions to capture irregular motion features. The number of output channels for each module gradually increases from 64 to 512, and a SE attention mechanism is added after convolution to enhance the response to key features. Cross-modal interaction layers are inserted after the third and fifth modules to fuse the spatiotemporal correlation between acceleration and angular velocity through a cross-attention mechanism.
[0121] Q and K originate from different sensor branches. A multi-scale adaptive pooling layer is added before the fully connected layer, combining 1x1, 3x3, and global average pooling scales, and dynamically weighting and fusing features of different granularities through a gating network. An uncertainty calibration unit is added before the Softmax layer, generating confidence weights based on Monte Carlo Dropout to suppress low-reliability predictions.
[0122] Building upon the existing adaptive learning rate, a hierarchical weight decay strategy is introduced, employing a decay coefficient of 0.001 for the bottom convolutional kernels and 0.0005 for the higher fully connected layers to avoid overfitting. Gradient clipping and Lookahead optimization are integrated, with parameter snapshots updated every 5 steps to improve stability. Meta-learning-driven hyperparameter tuning is employed, embedding a MAML framework to rapidly evaluate the model's generalization ability during cloud training using a small number of support set samples, dynamically adjusting the learning rate decay coefficient and batch size. A source-target feature decomposition module is built after the fully connected layers, utilizing a sliding window to statistically analyze the target domain prototype and achieving cross-domain feature alignment through the Wasserstein distance metric.
[0123] Where C i,j Let be the feature distance matrix between the source domain and the target domain.
[0124] An adaptive mechanism is employed to improve the model's robustness. This mechanism mainly consists of four parts: misclassification detection, error sample storage, trigger condition detection, and adaptive retraining. During the testing phase, as long as a prediction result is detected... Compared with the actual result (y) t If there are differences between them, misclassified data will be recorded.
[0125] Once the number of these errors (L) new The threshold (η) is reached, i.e., L new When the value is greater than η, the classifier will initiate an adaptive retraining process to improve its accuracy.
[0126] This enhanced architecture achieves high concurrency processing capabilities by deeply integrating structural complexity with adaptive mechanisms, while relying on cloud computing elastic architecture to meet real-time response requirements.
[0127] S5.3 Cloud Gauge Data Modeling
[0128] The fitted respiratory signal is obtained through multiple data processing steps. The main characteristics of the respiratory signal, such as frequency, amplitude, period, and waveform, are analyzed in the cloud through modeling.
[0129] Frequency analysis: Fourier transform is performed using a cloud-based distributed computing framework to convert the signal from the time domain to the frequency domain. Distributed computing can handle large-scale datasets and improve computational efficiency.
[0130] Clustering algorithms are used to automatically identify the main frequency components in the frequency spectrum, thereby determining the respiratory rate. The high computing power of the central control system supports the training and inference of complex models. Peak frequencies correspond to the main respiratory rates.
[0131] Amplitude analysis: Calculates the amplitude spectrum of the signal and uses a cloud-based distributed storage system to store intermediate results, ensuring high availability and reliability of the data. A i (f)=|S i (f)|
[0132] Analyze the amplitude of different frequency components to reflect changes in respiratory intensity. Utilize a cloud-based real-time data processing framework to monitor respiratory signals in real time, promptly detect abnormalities, and issue alerts.
[0133] Periodic analysis: By detecting the peaks or troughs of the respiratory waveform, the respiratory cycle, i.e., the time interval between two consecutive breaths, is calculated. Periodic analysis can reveal the rhythmicity of breathing and detect abnormalities such as sleep apnea. The period of the signal is calculated using peak detection methods. Let the period of the signal be T. i Then we have:
[0134] Among them, f peak This represents the peak frequency in the frequency spectrum. Periodic analysis can identify periodic changes in respiration and detect the rhythm of breathing.
[0135] Similarly, the intake time T is calculated using a peak detection algorithm. a and exhalation time T b And calculate the duty cycle.
[0136] Waveform analysis: The respiratory signal is decomposed into wavelet coefficients of different scales to analyze its waveform characteristics, such as rise time, fall time, and waveform morphology. Wavelet transform can identify more subtle changes in the respiratory waveform, such as abnormal breathing patterns like wheezing and shortness of breath. By analyzing the waveform characteristics of the signal using wavelet transform, we can further clarify the nature of the continuous wavelet transform of the signal. Let W be the wavelet transform of the signal. i (a, b), then we have:
[0137] Where a is the scaling parameter, b is the translation parameter, and ψ is the mother wavelet function. A deep learning model is used to extract detailed features from respiratory signals and identify abnormal breathing patterns. Cloud-based GPU resources can accelerate the training and inference of deep learning models.
[0138] S5.4 Cloud Data Comprehensive Analysis and Long-Term Storage
[0139] The cloud layer is not only responsible for data modeling, but also for comprehensive analysis of multi-source data and long-term data storage. The implementation methods for comprehensive analysis and long-term storage are described below.
[0140] S5.41 Cloud Data Comprehensive Analysis
[0141] Cloud data comprehensive analysis integrates multi-source data such as inertial navigation unit data and goniometer data to extract valuable insights to support decision-making and applications.
[0142] Statistical modeling is performed on multi-source data to analyze its distribution characteristics, correlations, and trends. Data visualization is implemented, displaying user health data trends through charts. Personalized health recommendations are generated based on the analysis results. If abnormal data is detected, the system will generate alarm messages and prompt users to take further health measures. Group health monitoring is conducted, combining data from multiple users to perform statistical analysis of group health status. This can be used to identify group health trends or the prevalence of certain diseases, providing data support for public health management.
[0143] The cloud layer supports real-time data analysis and utilizes stream processing frameworks to achieve low-latency analysis. Distributed computing resources ensure the scalability of analytical tasks, adapting to the growth of data volume.
[0144] S5.42 Cloud Data Long-Term Storage
[0145] At the cloud computing layer, the long-term data storage system employs a strategy combining a tiered storage architecture with automated lifecycle management to ensure optimal accessibility, security, and cost-effectiveness for massive amounts of medical data. Raw sensor data (such as ECG waveforms and respiratory flow time series) is ingested through the ` / batch / data` topic in Kafka, then first sharded and written to object storage (such as AWS S3) in columnar storage format (Parquet), and partitioned in a spatiotemporal dual-dimensional manner—by collection date in the time dimension (e.g., year=2023 / month=08 / day=14), and by user geographic location hash in the spatial dimension (e.g., region=us-east / bucket=05). This partitioning strategy significantly improves query efficiency by time range or geographic attribute, while avoiding file overload in a single directory.
[0146] To balance storage costs and access performance, the system introduces a three-tier storage strategy: hot-warm-cold.
[0147] 1. Hot storage layer: Retains data from the most recent 30 days and stores it in a distributed file system supported by SSD (such as HDFS) for high-frequency access and real-time analysis (such as retrieving a user's recent health trends).
[0148] 2. Warm storage layer: Data from 30 days to 1 year is migrated to a low-cost HDD cluster, using Erasure Coding (EC) redundancy strategy to reduce storage overhead by 40% while ensuring availability.
[0149] 3. Cold storage layer: Data older than 1 year is compressed into Zstandard format (compression ratio up to 5:1) and archived to tape library or Glacier-like services. It only supports offline batch reading and is suitable for compliance audits or long-term scientific research analysis.
[0150] The data lifecycle is dynamically managed by a policy engine: based on access pattern analysis (such as metadata access logs recorded by Apache Atlas), low-frequency access data is automatically downgraded for storage; simultaneously, retention periods are set according to data sensitivity (such as ECG data containing personal health information) (such as the 6 years stipulated by the EU GDPR), triggering an automated erasure process upon expiration. The erasure process records tamper-proof audit logs to blockchain nodes for compliance traceability. The cloud layer manages metadata uniformly through a global data catalog (such as AWS Glue), labeling each data file with information such as the source device, acquisition parameters, and preprocessing version, and linking with the permission system (such as Apache Ranger) to achieve column-level fine-grained access control (such as allowing only cardiologists to access abnormal heart rate records of specific patients). The decoupling design of the storage layer and computing resources (such as performing S3 direct lookups via Presto) eliminates the need for data migration for data analysis, further reducing operational complexity.
[0151] As shown in Figure 2, the Kafka topic partitioning design (partition 1 to partition n) supports horizontal scaling, allowing the number of partitions to be dynamically increased based on data throughput, avoiding single-point bottlenecks. The user storage layer uses multi-tenant isolated storage for "topic: user 1-user n," ensuring granular control of data permissions. Goniometer data at the edge layer employs adaptive Kalman filtering to fuse multi-sensor data, adjusting weight coefficients in real-time to reduce motion interference. The fog layer uses a lightweight convolutional network, achieving low-latency motion recognition through model pruning, and is regularly updated in sync with the cloud-trained model. ECG data is used for QRS wave localization in the fog layer, and Flink's Event Time mechanism is combined to process out-of-order data streams, ensuring medical-grade time-series accuracy.
[0152] Noise reduction of inertial sensor data is performed at the edge layer, reducing the amount of raw data and lowering the computational resource consumption of the fog layer. When an edge layer node fails, partitions are automatically reallocated. Combined with Flink's Savepoint mechanism, seamless continuation of processing offsets is achieved with zero data loss. Cloud storage connects to the Flink Batch API, supporting offline analysis of historical data without affecting real-time stream processing throughput, enabling mixed T+0 and T+1 queries.
[0153] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture, characterized in that, include: Acquire sensor data, including inertial navigation unit data acquisition, goniometer data acquisition, and electrocardiogram data acquisition. Synchronize the acquired sensor data signals; The synchronized data signals are transmitted in layers, with sensor data processing performed sequentially at the edge layer and fog layer before being transmitted to the cloud layer; sensor data modeling and comprehensive analysis are then performed in the cloud layer. Store the results of data analysis and processing.
2. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 1, characterized in that, The data signal synchronization of the acquired sensor data includes data synchronization for multiple sensors of the same type during transmission, including data transmission synchronization of multiple inertial navigation units and data transmission synchronization of multiple goniometers. For the inertial navigation units, the receiving time is captured by an event timer, the propagation delay is calculated, and bidirectional delay measurement is used to eliminate channel asymmetry error. The integrity of the synchronized data is verified by fusion based on a Kalman filter. For the goniometers, the linear relationship between the master and slave clocks is fitted by the least squares method, and nonlinear correction is performed based on clock deviation compensation.
3. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 2, characterized in that, The sensor data processing is performed sequentially at the edge layer and the fog layer. This includes inertial navigation unit (INS) data processing at the edge layer and classification of human dynamic and static movements based on a set threshold using INS data. Specifically, the INS data is denoised using a Kalman filter algorithm for each dimension of the signal. The system then predicts the next time step, predicts the state covariance based on the state transition matrix, and calculates the Kalman gain K based on the predicted covariance and the observation noise covariance. k To balance predicted and observed values, the calculated Kalman gain K is used. k Combined with the observed value Z k Update the predicted state to obtain the optimal estimated state at time k. At the same time, update the optimal estimated covariance.
4. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 3, characterized in that, The process of processing sensor data sequentially in the edge layer and fog layer also includes processing goniometer data in the edge layer. Specifically, the square root of the two channels of each data segment of a single goniometer is calculated, the signals of the two channels are merged, the continuous respiratory signal data is divided into multiple data segments according to time periods, a composite respiratory signal is generated by using an energy fusion algorithm on multiple goniometer data streams, and the dynamic weight SNR of each goniometer signal within the sliding window is calculated.
5. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 4, characterized in that, The process of processing sensor data sequentially in the edge layer and the fog layer also includes processing electrocardiogram (ECG) data in the edge layer. Specifically, baseline drift correction is performed on the acquired ECG data, and a high-pass Butterworth filter is used to separate baseline drift below 0.5 Hz. Then, the QRS complex is enhanced by using a bandpass filter to specifically enhance the QRS wave energy and suppress electromyographic noise of 20-500 Hz and low-frequency T-wave components below 5 Hz.
6. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 5, characterized in that, The sensor data processing is performed sequentially at the edge layer and the fog layer, including inertial navigation unit data processing in the fog layer, and training and classifying static motions using a DCNN model. Specifically, the acceleration data of the inertial navigation unit is processed to mitigate the influence of gravity on motion classification, as shown below: Where H(s) represents the transfer function, s represents the complex frequency variable in the Laplace transform, and Q... k The quality factor is ω0, and the center angular frequency is ω0. These are filtered triaxial acceleration data. It is the acceleration data after removing the influence of gravity.
7. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 6, characterized in that, The process of processing sensor data sequentially in the edge layer and fog layer also includes processing goniometer data in the fog layer. This involves using a hybrid filtering architecture to denoise the fused goniometer data, dynamically adjusting the time window length, prioritizing the retention of low-frequency components of the respiratory signal, and employing a Butterworth filter with a cutoff frequency of 10Hz to eliminate high-frequency interference from muscle tremors. After denoising, the data undergoes baseline correction and normalization, morphological baseline extraction is used, and opening and closing operations are performed using structuring elements matched to the respiratory cycle to separate slowly changing baseline drift. Respiratory dynamics analysis is then performed to reveal the hidden dynamic characteristics of respiration through respiratory signals. The maximum Lyapunov exponent is calculated to measure the sensitivity of the respiratory cycle to initial conditions; a higher value indicates a more unstable rhythm.
8. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 7, characterized in that, The sequential sensor data processing at the edge layer and fog layer also includes ECG data processing at the fog layer. Specifically, QRS peaks in the ECG data are detected by introducing a dynamic integration window into the Pan-Tompkins algorithm, allowing the window length to automatically adjust with heart rate and avoiding waveform overlap at high heart rates. Then, after performing squaring and moving integration, time-frequency domain features are extracted from the ECG signal, and the continuous RR interval sequence {Δt} is calculated. i The power spectral density was estimated using the Lomb-Scargle algorithm, and finally, respiratory-electrocardiogram coupling analysis was performed to establish a phase synchronization model between the respiratory signal D_norm(t) and the RR interval.
9. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 8, characterized in that, The aforementioned sensor data modeling and comprehensive analysis in the cloud includes inertial navigation unit (INS) data modeling in the cloud. INS data modeling utilizes a DCNN classifier and employs six progressively increasing modules. The first three modules use a residual Inception structure, while the latter three modules introduce dynamically deformable convolutions to capture irregular motion features. The number of output channels for each module gradually increases from 64 to 512. An SE attention mechanism is added after convolution to enhance the response to key features. Cross-modal interaction layers are inserted after the third and fifth modules to fuse the spatiotemporal correlation between acceleration and angular velocity through a cross-attention mechanism.
10. The method for real-time analysis of multimodal wearable sensor data based on a hierarchical architecture according to claim 9, characterized in that, The aforementioned sensor data modeling and comprehensive analysis in the cloud also includes goniometer data modeling in the cloud. This involves using a distributed computing framework to perform Fourier transform, converting the signal from the time domain to the frequency domain, and using clustering algorithms to automatically identify the main frequency components in the frequency spectrum to determine the respiratory rate. The amplitude spectrum of the signal is calculated, and intermediate results are stored using a cloud-based distributed storage system. The respiratory cycle, i.e., the time interval between two adjacent breaths, is calculated by detecting the peak or trough values of the respiratory waveform. The respiratory signal is then decomposed into wavelet coefficients of different scales to analyze the waveform characteristics of the signal, and the data is comprehensively analyzed and stored long-term.