Human activity recognition using distillation, augmentation, and filtering

Abstract

Systems and methods for human activity recognition. An example method includes receiving temporal sensor data within an interval of sensor measurements, processing, with an electronic processor, the temporal sensor data through a first student model to obtain a first activity class probability, and processing, with the electronic processor, the temporal sensor data through a second student model to obtain a second activity class probability. The example method also includes performing, with the electronic processor, probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability. The example method also includes outputting, with the electronic processor, a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

Description

D24040W001HUMAN ACTIVITY RECOGNITION USING DISTILLATION, AUGMENTATION, AND FILTERINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from Indian Provisional Application No. 202511018133 filed on 28 February 2025, and European Application No. 25181966.0 filed on 11 June 2025, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] This application relates generally to human activity recognition using sensors, and more specifically but not exclusively, to training and deploying machine learning models for human activity classification.BACKGROUND

[0003] Herein, the term “Human Activity Recognition” (HAR) refers to a process of using machine learning algorithms and sensor data from one or more devices to detect and categorize various human activities, such as sitting, standing, climbing stairs, walking, running, cooking, etc. HAR is an active area of research, for example, in ubiquitous computing, human behavior analysis, and human-machine interaction. Example applications of HAR include maintaining a healthy lifestyle, detecting progression of illnesses, gaming, and many more.SUMMARY

[0004] Embodiments of the present disclosure provide a multiple-architecture-based multi-teacher framework for performing knowledge distillation. Contrastive learning-based loss is employed for performing smartphone based efficient HAR with multi-variate time-series sensor data. Some examples employ contrastive loss, focal loss, generative modeling based synthetic data augmentation, and mix-up-based augmentation.

[0005] Some embodiments of the present disclosure provide a probability filtering technique. The probability filtering technique equips the multiple-architecture-based multi-teacher framework with a capability to fuse the decisions of two student variants that can be used in parallel for inference,D24040W001while keeping the inference latency substantially the same as or close to as that of a single lightweight base student network.

[0006] According to embodiments of the present disclosure, a method for human activity recognition comprises receiving temporal sensor data within an interval of sensor measurements; processing, with an electronic processor, the temporal sensor data through a first student model to obtain a first activity class probability; processing, with the electronic processor, the temporal sensor data through a second student model to obtain a second activity class probability; performing, with the electronic processor, probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability; and outputting, with the electronic processor, a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

[0007] According to embodiments of the present disclosure, a method for training a human activity recognition system comprises training a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture; identifying, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data; training a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model; and training a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.

[0008] According to embodiments of the present disclosure, an apparatus for human activity recognition comprises at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive temporal sensor data within an interval of sensor measurements; process the temporal sensor data through a first student model to obtain a first activity class probability; process the temporal sensor data through a second student model to obtain a second activity class probability; perform probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability; and output a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

[0009] According to embodiments of the present disclosure, a non-transitory computer-readable medium is described. The non-transitory computer-readable medium storing instructions that, whenD24040W001executed by an electronic processor, cause the electronic processor to perform operations comprising receiving temporal sensor data within an interval of sensor measurements; processing, with an electronic processor, the temporal sensor data through a first student model to obtain a first activity class probability; processing, with the electronic processor, the temporal sensor data through a second student model to obtain a second activity class probability; performing, with the electronic processor, probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability; and outputting, with the electronic processor, a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

[0010] According to embodiments of the present disclosure, an apparatus for human activity recognition comprises at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to train a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture; identify, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data; train a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model; and train a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 illustrates an example of a mobile device according to embodiments of the present disclosure.

[0012] FIG. 2 illustrates a distillation-augmentation-filtering pipeline for human activity recognition according to embodiments of the present disclosure.

[0013] FIG. 3 is a schematic diagram illustrating a detailed example of the knowledge distillation stage according to embodiments of the present disclosure.

[0014] FIG. 4 illustrates an example of a variational autoencoder for performing data augmentation according to embodiments of the present disclosure.

[0015] FIG. 5 illustrates a checkpoint ensembling process according to embodiments of the present disclosure.D24040W001

[0016] FIG. 6 illustrates an example workflow for human activity recognition according to embodiments of the present disclosure.

[0017] FIG. 7 illustrates an example workflow for training a human activity recognition system according to embodiments of the present disclosure.

[0018] FIG. 8 illustrates an example apparatus for human activity recognition according to embodiments of the present disclosure.DETAILED DESCRIPTION

[0019] Human Activity Recognition (HAR) using smartphones is an application of mobile computing that leverages the rich array of the smartphones’ built-in sensors, such as accelerometers, gyroscopes, and magnetometers. These multiple types of sensors generate multivariate time-series data. The collected sensor data may capture temporal patterns of human activities. In some examples, HAR have evolved from simple heuristic-based methods to machine learning techniques, with deep learning models becoming increasingly prevalent due to the ability to learn complex patterns from raw sensor data.

[0020] Machine learning techniques may be used in smartphone-based HAR. In some examples, deep learning models are computationally intensive for real-time deployment on mobile devices with limited resources. In some examples, such models may have difficulties in maintaining consistent performance across different activity classes, particularly for activities with limited training data or those that are difficult to distinguish from similar activities. In some examples, knowledge distillation techniques may be employed to transfer knowledge from large models to smaller ones. However, in some cases, these methods may not be sufficiently effective in capturing class-specific expertise from multiple teacher models. In some examples, such methods may lack effective mechanisms for combining predictions from multiple models, thereby causing underperformance when attempting to leverage complementary strengths of different models.

[0021] At least some embodiments disclosed herein address these challenges through several new and / or improved features. In some aspects, embodiments of the present disclosure introduce a multiarchitecture teacher framework where different teacher models specialize in different activity classes, thereby enabling more effective knowledge transfer to lightweight student models. In some aspects, embodiments of the present disclosure implement a contrastive learning approach for knowledge distillation that helps student models learn not only what features to emphasize but also what features to avoid for each activity class. In some aspects, embodiments of the presentD24040W001disclosure employ a dual-student architecture where one student model focuses on general activity recognition while a second student specializes in challenging cases through targeted data augmentation. In some aspects, embodiments of the present disclosure introduce a probability filtering mechanism that selectively combines predictions from both student models based on their validated strengths for each activity class, which tends to improve the performance while maintaining computational efficiency for mobile deployment.

[0022] FIG. 1 is a block diagram illustrating a mobile device 100 according to some examples. In the example shown, the mobile device 100 includes a proximity sensor 105, an ambient light sensor 110, a CMOS imager sensor 115, a GPS sensor 120, an accelerometer 125, a humidity sensor 130, a microphone 135, a touch sensor 140, a gyroscope 145, a magnetometer 150, a fingerprint sensor 155, a pressure sensor 160, and a temperature sensor 165. By having an array of multiple sensors, the mobile device 100 may perform tasks including classifying and recognizing human activities and gestures based on data collected via the multiple sensors. The data collected by these smartphone sensors may be used to detect various human movements.

[0023] The sensor arrays collect data which are multivariate time-series signals in nature. This characteristic arises because various types of hardware sensors (e.g., accelerometer 125, gyroscope 145) consist of multiple axes for data collection (e.g., x-, y-, z-axis). Sensor-driven Human Activity Recognition (HAR) is based on the formalized study of detecting human movements via data acquired from these smartphone sensors. HAR may include specifying a certain duration of time (commonly called a window) for which a part of the collected signal is considered, filtered, discretized, and then recognized by an underlying logic. This logic could be a heuristic observation made on the x-, y-, or z-axis values of the signal or an algorithm. In some examples, for simple movements made in an idealistic data collection scenario (e.g., an ideal, well-defined pose of the human), it may suffice to consider rules made on the values for recognizing an activity. However, such ideal collection scenarios are seldom found in practice, especially in complex human movements.

[0024] The requirement for the logic to accurately detect arbitrary, noisy movements on-the-fly presents significant technical challenges. Beyond handling noisy collection environments, there may be device delays and lags. In some examples, in the entertainment or gaming sector, users may not perform movements exactly as expected. These challenges restrict the capability of simple rulebased heuristics to serve as effective logic for detecting activities within a signal window. In these examples, Deep Learning (DL) comes into play. DL models are a class of neural network based,D24040W001Machine Learning (ML) model variants that effectively learn various complex correlations and patterns present in sensor data to make accurate predictions. Given adequate and diverse representative data, DL models for HAR can be used for recognizing complex human motions with higher Degrees of Freedom (DOF). For example, using DL models for HAR can detect movements with variations or noise that may cause rule-based methods to fail.

[0025] The multiple sensors shown in FIG. 1 may be used to collect data for the human activity recognition framework presented in subsequent figures. While these sensors provide rich multivariate time-series data, one of the challenges lies in efficiently processing and learning from the data in resource-constrained mobile environments. The following figures illustrate approaches that may be used to address various challenges. FIG. 2 shows an overall distillation-augmentation-filtering pipeline that enables efficient processing of sensor data. FIG. 3 details the knowledge distillation mechanism for transferring expertise from complex teacher models to lightweight student models. FIG. 4 shows the architecture for generating synthetic training data to improve recognition of challenging activities. FIG. 5 illustrates a probability filtering method that combines predictions from multiple student models.Distillation-Augmentation-Filtering (DAF) Pipeline

[0026] FIG. 2 illustrates a distillation-augmentation-filtering pipeline 200 for human activity recognition. The distillation-augmentation-filtering pipeline 200 is a multi-stage framework that includes a knowledge distillation stage 205, a data augmentation stage 210, and a probability filtering stage 215.

[0027] In FIG. 2, the knowledge distillation stage 205 operates to transfer knowledge from multiple teacher models to a student model. For example, the knowledge distillation stage 205 is configured to use contrastive learning to align similar teacher distributions and separate dissimilar teacher distributions, along with focal loss for handling difficult training samples. Herein, the terms "teacher" model and "student" model refer to machine learning models in a system that employs a knowledge transfer training approach, where relatively complex neural network models (e.g., the teacher models) with high accuracy but high computational cost train simplified neural network models (e.g., the student models) that maintain a level of accuracy while incurring fewer computing resources for deployment.

[0028] Some conventional machine learning models may exhibit relatively high latency that may be unsuitable for real-time inference. The knowledge distillation stage 205 addresses this issue byD24040W001implementing a multiple-architecture teacher framework where each activity class has an associated best-performing teacher model.

[0029] In some examples, the knowledge distillation stage 205 operates in two phases including a pre-distillation phase and a distillation phase. In the pre-di still ation phase, the knowledge distillation stage 205 trains multiple supervised model architectures and identifies a best-performing teacher model for each activity class based on validation data performance. During the distillation phase, the knowledge distillation stage 205 processes each training example by taking an associated or a respective class-specific model as a positive teacher and taking models associated with other classes as negative teachers. For example, the knowledge distillation stage 205 performs class-specific selection based on class-specific selection indicators. The class-specific selection indicators may be validation data-based metrics that determine which student model’s prediction is selected for each activity class during inference, based on each student model’s demonstrated performance for that specific class.

[0030] In the knowledge distillation stage 205, a real window 220 comprises a fixed-duration interval of raw sensor measurements collected from device sensors including accelerometers, gyroscopes, and cameras. The fixed-duration interval of raw sensor measurements may capture temporal patterns of user activities in a multi-dimensional time series format suitable for activity recognition processing.

[0031] When performing knowledge distillation, there may be an optimization burden on a single student model to both learn effectively during distillation and maintain robustness across imbalanced and confusing or challenging classes. The data augmentation stage 210 addresses the optimization burden by training a parallel student model for an original student model, where the parallel student model is trained to focus on challenging classes during the knowledge distillation. In this way, the parallel student model and the original student model complement each other’s performance capabilities.

[0032] In some examples, the first student model 235 implements a computationally efficient neural network architecture trained through knowledge distillation. Training the first student model 235 may involve using cross-entropy loss to maintain classification accuracy and distillation loss to incorporate knowledge from the teacher model set 230, training a lightweight model to be suitable for mobile deployment.

[0033] The data augmentation stage 210 may generate additional training data for challenging activity classes using synthetic data generation and Mix-Up techniques. The Mix-Up techniquesD24040W001refer to techniques that combine pairs of training examples and corresponding labels in varying proportions to create new synthetic training data. By using the Mix-Up techniques, machine learning models leam more robust decision boundaries between classes.

[0034] In the data augmentation stage 210, an augmented window 225 comprises augmented data that mirrors the temporal and multi-dimensional characteristics of real window measurements. The augmented data may be synthetically generated sensor measurement data. The augmented data may be generated by the generative / augmentation model 245 in FIG. 2 through techniques such as synthetic time-series generation and Mix-Up data combination, with focus on challenging activity classes where additional training examples are needed.

[0035] In some examples, the teacher model set 230 contains multiple neural network models, each implemented with distinct architectural designs for activity recognition. In some instances, during pre-distillation training, teacher model set 230 processes validation data to establish performance metrics for each activity class, enabling identification of the most effective teacher model architecture for each specific class of activity. The data augmentation stage 210 then trains a second student model 240 using this augmented data in combination with the original training data.

[0036] The data augmentation stage 210 involves training the second student model 240 based on the generative / augmentation model 245 and the augmented window 225. The second student model 240 may be a machine learning model that implements a parallel student neural network architecture trained on both original and synthetically augmented data. Training the second student model 240 may involve combining cross-entropy loss for classification accuracy, distillation loss for teacher knowledge transfer, and augmentation loss to handle challenging activity classes, resulting in a complementary lightweight model focused on difficult cases.

[0037] The probability filtering stage 215 implements a checkpoint ensemble mechanism to combine predictions from both student models. In some instances, after obtaining the trained student model checkpoints, each specialized for different strengths, the probability filtering stage 215 applies class-specific selection indicators based on validation performance to determine which student model's prediction to use for a respective class.

[0038] In some examples, the checkpoint ensemble is a validation-driven model selection mechanism that determines a desirable combination of the two student models' outputs. For example, the checkpoint ensemble process evaluates each student model's performance on validation data to establish class-specific selection indicators, which are then used during inference toD24040W001determine which student model's predictions should be used for each activity class in the final probability distribution.

[0039] In some examples, the probability filtering stage 215 processes validation data through both the trained first student model and the trained second student model to determine class-specific selection indicators. The probability filtering stage 215 then applies the class-specific selection indicators to select the student model predictions for each activity class when generating the probability distribution. In some instances, the probability filtering stage 215 processes logit information independently in the post-training phase, separate from the joint logit processing performed during distillation.

[0040] According to some aspects, at the pre-distillation stage of the multi-stage framework illustrated in FIG. 2, multiple supervised classifier model architectures are trained, and a best performing model corresponding to each of the classes is identified. For each classifier, this may be conducted with the cross-entropy term £CE. Then, a distillation of the consolidated information of the teachers into a single student model is performed, using £CEand a distillation term £aistu- Both the pre-distillation training of teachers, and distillation of the student is performed with the multivariate real time-series data. Next, the data augmentation is performed to focus on challenging classes. In this example, the augmentation can be performed using a synthetic data generation technique based on generative modelling, or by an augmentation technique such as the Mix-Up technique. In some examples, a loss term £augis employed to focus on the augmented data, in addition to performing distillation for the second student model 240. The decisions made by the two student variants are fused together in the probability filtering stage 215.Knowledge Distillation

[0041] FIG. 3 is a schematic diagram illustrating the knowledge distillation stage 205 according to some examples. In the example shown, the knowledge distillation process begins with an input time window Xi with ground truth label— c, which is processed by a lightweight student model to generate logits ■$■(%,). These logits, along with those generated by positive teacher 305 (Mc) and negative teacher 310 (Md), are each transformed through softmax functions with temperature parameter T to produce probability distributions. The contrastive learning process 315 then orchestrates the knowledge transfer by simultaneously using cross-entropy loss to align the student's predictions with the ground truth, implementing a "pull" mechanism to draw the student's distribution closer to the positive teacher's distribution for the correct class c, and executing a "push" mechanism to distance the student's distribution from the negative teachers' distributions for allD24040W001other classes d. This coordinated process, governed by contrastive distillation loss that combines both pull and push operations, trains the lightweight student model to learn from multiple teacher models while maintaining proper class discrimination.

[0042] The positive teacher 305 (Mc) represents the teacher model specifically associated with the target class c for a given input window x,. The positive teacher 305 generates logits Mc(Xj) which are processed through a softmax function with temperature parameter T to produce a probability distribution. This probability distribution is a target that the student model aligns with during the knowledge distillation process.

[0043] The negative teacher 310 (Md) encompasses the teacher models associated with classes other than the target class c. For each class d in the set {1,2,..., C]\{c}, the negative teacher 310 generates logits Md(Xj), which are processed through the same softmax function to produce probability distributions that the student model should learn to differentiate from.

[0044] The contrastive learning process 315 may implement a distillation mechanism that simultaneously pulls the student model’s distribution closer to the positive teacher's distribution while pushing it away from the negative teachers' distributions. The contrastive learning process 315 may utilize a loss function that combines both attractive (pull) and repulsive (push) forces to shape the student model's learning. For example, an InfoNCE-based loss function may be used, where the Info stands for information and the NCE stands for Noise-Contrastive Estimation. The InfoNCE-based loss function is an example of a loss function used for self-supervised learning. The contrastive learning process 315 employs focal loss to handle training examples of varying difficulty levels, with a modulating factor that down-weights well-classified examples.

[0045] The detailed example of the knowledge distillation stage 205 illustrated in FIG. 3 may be realized using the following example operation. In the example operation, the overall interaction between these components is governed by a total loss function that combines cross-entropy loss (for ground truth alignment) and contrastive distillation loss (for knowledge transfer), weighted by a hyperparameter a to balance their contributions to the student model's training.

[0046] A loss formulation to perform knowledge distillation can be expressed as:^total =a£cs + (1 - (!) Here, LCErepresents the cross-entropy loss term which tries to align the model prediction with that of the ground-truth represented as a one-hot representation. LCEis expressed as:D24040W001ℒCE= (2)i=1 j=1

[0047] Here, xi∈ ℬ, is a training example in a mini-batch ℬ, belonging to the class c ∈{1,2,..., C}. The ground truth class label is provided as a one-hot representation yi∈ ℝC, such that Yi(c) = 1, and yi(d) = 0, if d ≠ c. In short-hand integer notation, one can also represent ground truth label as an integer, as yi= c. |B| is the number of examples in the mini-batch, used to normalize the loss. The meaning of o(s(Xi); T)(j) will be explained shortly, along with the distillation loss term.

[0048] ℒstddistilis the distillation loss term which facilitates transfer of the “dark knowledge” from the teacher model to the student model. In machine learning, "dark knowledge" refers to the additional, nuanced information captured by a large, complex "teacher" model that goes beyond just the correct prediction, which can be transferred to a smaller "student" model through a process called knowledge distillation. For example, the dark knowledge represents the subtle relationships and patterns learned by the teacher model that are not explicitly stated in the labels, but can be valuable for improving the student model's performance. Here, T > 1 is the temperature parameter which controls the smoothness of the distribution via the soft-max function. The hyperparameter a > 0 is used to balance the two loss terms while computing the total loss ℒtotal. The distillation loss term is computed as:σ(Mfixed(xi); τ), σ(s(xi); τ)) (3)

[0049] The loss term given by Eq. (3) is used to substantially minimize the distance) between the target distribution o’(Mxed(xi); T) provided by the fixed teacher model Mfixed. ), and the predicted distribution σ(s(xi); τ) by the student model s(.). In some examples, ℋstddist(.) is computed using the Kullback-Leibler divergence (DKL), which, for a respective pair of target and predicted distributions p(x) and q(x) (where x G X, the sample space), can be expressed as:ℋstddist(.) = DKL(p(x)||q(x)) = ∑x∈풳p(x) log (4)x∈풳

[0050] For a training example, x, G B, in a mini-batch B, and a function z(. ) parameterizing a neural network model, the logits could be denoted as z(Xj), and can be used to represent without loss of generality, both teacher logits Mfixed(xi), as well as the student logits s(xi).D24040W001

[0051] Assuming that there are C classes (1, 2,..., C) and that c denotes the cth class, embodiments of the present disclosure can make use of the soft-max function cr(.; T) with the temperature parameter T to compute the probability of x, being predicted as belonging to the cth class as follows:exp(z(xi)(c) / τ) pi(c) = σ(z(xi); τ)(c) =∑Ck=1exp(z(xi)(k) / τ) Here, pi∈ ℝCis the output probability distribution for the training example xiand Pi(c) is the cth component of vector p,.

[0052] To extend the above formulation to a multi-teacher setup, for each class c, embodiments of the present disclosure can have an associated teacher Mc(. ) which performs the best among all the candidate model architectures being tested. This identification can be done by assessing the comparative performance of all the supervised classifiers from the pre-distillation stage. Then, embodiments of the present disclosure can use a Class-Specific Activation (CSA) strategy to perform the distillation as follows:tℒcsadistil= ℋstddist(σ(Mc(xi); τ), σ(s(xi); τ)) (6)Here, the teacher Mc(.) associated with class c may be invoked to perform distillation for xisuch that yi= c, where yidenotes the class label for xi.

[0053] In some instances, while the above CSA strategy for distillation considers the positive teacher Mc(. ) for the class c, it does not consider any other teacher from the set of negative teachers {Md(. )}, Vd G {1,2, •••, C} \ {c}. The model may thus be unable to learn about what is repelled away because of the set of negative teachers. To address this, the InfoNCE-based Contrastive Learning (CD), may be employed, which not only helps to teach a model about what needs to be brought closer, but at the same time teaches what needs to be dissimilar.

[0054] In the context of this example operation, according to aspects of the present disclosure, the distillation loss term is modified to perform InfoNCE-based Contrastive Learning. The contrastive distillation is expressed as:ℒcddistil= (1 / |ℬ|) ∑x∈ℬlog [exp(ℋci(xi)) / ∑d≠cexp(ℋdi(xi))] (7)∑d≠cexp(ℋdi(xi)) Here, ℋci(xi) is expressed as:D24040W001ℋci(xi) = ℋdist(σ(Mc(xi); τ), σ(s(xi); τ)) (8)

[0055] The division by |B| is for normalizing the loss term. ℒcddistilis employed to align the student model prediction distribution σ(s(xi); τ) for an example Xj closer to that of the distribution σ(Mc(xi); τ) obtained for the positive teacher model associated with the class c = yi. At the same time, it ensures learning from the negative teachers as well, by simultaneously pushing away the distributions σ(Md(xi); τ), ∀d ∈ {1,2, ···, C} \ {c}.

[0056] To further make the loss term more robust across examples of varying levels of difficulty in terms of being predicted correctly, examples of the present disclosure use the Focal Loss FL(pt) in the J-Cdist ) term, computed as:ℋdist= FL(pt) = −αt(1 − pt)γlog(pt) (9)

[0057] Here, atG [0,1] is a weighting factor to for handling class imbalance, and can be set by inverse class frequencies. However, despite its presence, the gradient may be dominated by the loss terms corresponding to easily classified negative examples. To address this, embodiments of the present disclosure introduce the modulating factor (1 — pt)y. Here, as pt-> 1, i.e., an well-classified example, the loss term is down-weighted by y. Accordingly, the total loss as can be computed as:Fℒtotal= αℒCE+ (1 − α)ℒcddistil(10)

[0058] In some instances, as compared to using τ2ℒstddistilas used earlier to compute ℒtotalin standard distillation, the training may not benefit from multiplying contrastive distillation ℒcddistilwith τ2, although both ℒCEand ℒcddistilmake use of τ. Accordingly, embodiments of the present disclosure avoid this problem in the above computation of ℒtotalby using contrastive distillation.Data Augmentation

[0059] According to some aspects, the performance of the student model is dependent on various factors besides the knowledge distillation. For example, while the temperature t controls the smoothness of the teacher distribution, and thus the dark knowledge, the capability gap among the teacher and student architectures also plays a role. In some instances, the student and teacher models belong to a similar family of architectures. Embodiments of the present disclosure are leveraging multiple teacher architectures, and hence, the gap in capabilities of the student and various teachers differ significantly. In some example experiments, embodiments of the present disclosure observedD24040W001that our framework benefitted from using larger values of T, which bridges the capability gap between the student and teacher architectures and leads to an effective distillation.

[0060] In some instances, the overall performance is dependent not only on the DL model architectures, but also on the data quality and quantity. Because there may be data imbalance in HAR data, embodiments of the present disclosure may encounter some classes that are underrepresented in terms of the number of examples for those classes. At the same time, there may be inherently confusing pairs of classes in the data. It may not be feasible in practice to collect more data to alleviate this problem. For instance, for an application where a user creates their own hand movement classes on-the-fly, in-the-wild, larger data sets to train (or fine-tune) upon may not be obtained. There are also instances where the data is noisy. Therefore, being able to learn from limited data, in few-shot settings with a couple of windows collected for a class is desirable and a challenge.

[0061] To address the above challenges, embodiments of the present disclosure suggest utilizing two forms of data augmentation including (i) Synthetic data generation by using generative modeling, and (ii) Mix-Up based augmentation.Data Augmentation: Synthetic Data Augmentation Using Generative Modeling

[0062] Generative methods are used in domain-specific applications (e.g., computer vision, audio, text, etc.). These domain-specific applications are different from time-series data generation. An example of the generative methods is by using the Generative Adversarial Networks (GANs). The GANs can generate high-quality data by using a pair of Generator and Discriminator networks, where the Generator network synthesizes fake samples that mimic real data, and the Discriminator network is trained to distinguish real samples with that of fake ones. The training involves an adversarial tactic and may be considered as unsupervised without requiring class labels for a sample.

[0063] Another example of the generative methods is by using the Variational Auto-Encoders (VAE), which is also an unsupervised approach. The VAE has an Encoder network and a Decoder network, and can be used to generate synthetic data. VAE compresses a real data sample into a latent code using the Encoder, while capturing its features. The Decoder then aims at reconstructing the original data from the latent representation. A reconstruction loss is used to perform the operation. In the process, the model learns the underlying data distribution which can be used to generate newer samples.

[0064] Another example of the generative methods is by using diffusion models. The diffusion models add noise to real data samples gradually, to corrupt them step-by-step. Then, the reverse mapping is learned, to obtain clean data from the noisy ones. This process also teaches the model theD24040W001underlying data distribution which can be used to generate newer samples. In domains like computer vision, diffusion models have been used to generate sharper quality samples.

[0065] There are some challenges with using these generative methods. For example, the diffusion models, despite their high-quality sampling, are iterative, slower, and computationally expensive. The diffusion models also require larger amount of data. The GANs on the other hand suffer from model collapse, resulting in lesser diversity in the samples generated. Unstable behavior is also observed when training GANs for the application described herein. The VAE approach is demonstrated to be more stable during training and more capable of generating more realistic looking samples.

[0066] According to aspects of the present disclosure, in some examples, the augmentation step may be generic in nature and can accommodate off-the-shelf SOTA synthetic data generation method. In some alternative examples, a VAE is employed for the augmentation step to provide a relatively fast and stable training with reasonably realistic generated data. In some instances, for the augmentation step, the classes that are underrepresentation and the classes that may be confused with other classes are identified and assigned with more attention during the training. This identification may also be guided using the general performance of the supervised teacher models in the pre-distillation phase, along with the performance of the distilled student model. When such a class is identified, embodiments of the present disclosure train an unsupervised VAE specific to that class.

[0067] FIG. 4 illustrates an example of a variational autoencoder (VAE) 400 for performing data augmentation according to embodiments of the present disclosure. The variational autoencoder 400 may be used to perform data augmentation by generating synthetic data. In FIG. 4, the variational autoencoder 400 comprises an encoder 405 and decoder 410 configured to generate synthetic timeseries data for data augmentation. The encoder 405 transforms input time-series data X through a series of one-dimensional convolutional layers (ConvlD), followed by flatten and dense layers. The encoder outputs two parameters: a mean ( / / ) and variance (cr) that characterize a multivariate Gaussian distribution. This distribution represents a learned compressed representation of the input data.

[0068] The decoder 410 takes a sample Z drawn from this learned Gaussian distribution and reconstructs an approximation of the original input data X. The decoder's architecture mirrors the encoder in reverse, using one-dimensional transpose convolutional layers, reshape operations, and dense layers to progressively transform the latent sample back into the time-series data format.

[0069] The architecture employs a specialized loss function that combines reconstruction error with a Kullback-Leibler divergence term. By using the Kullback-Leibler divergence term, the encoder learnsD24040W001to map the input data to a well-structured latent space that follows a standard normal distribution, while the decoder learns to generate realistic synthetic data samples that preserve the statistical properties of the original time-series data.

[0070] In some examples, the loss function for the VAE can be expressed as:^ℒθ,φ= − 피q(z|x;φ)[log pθ(x|z)] + KL(q(z|x; φ) || pθ(z)) (11)

[0071] Here, in the RHS, the first term is the negative log-likelihood of our original data given variable z generated from some unknown prior distribution pθ(z). The decoder network serves to model pθ(x|z). The encoder serves to model a posterior distribution q(z|x; φ) which seeks to approximate the expensive pθ(z). In some instances, pθ(z) is chosen to be the standard Normal distribution. θ and φ are the respective learnable parameters. By using the KL divergence term embodiments of the present disclosure try to regularize the posterior to ensure the wrapping of the latent space around the prior distribution. By doing so, embodiments of the present disclosure are able to achieve our broader goal of generating synthetic samples that reasonably match the original data distribution by modeling an unknown probability function p(x).

[0072] According to some aspects, the synthetic time series data generated using a class-specific VAE trained on an underrepresented class demonstrates a similarity with the sample real time series. Side-by-side comparison of data acquired by different sensors for a pair of confusing classes in our data set. The rows correspond to the two classes, and the columns correspond to the sensor types.Data Augmentation: Mix-Up Augmentation

[0073] In some examples, the generative method based synthetic data augmentation involves a separate training step using a specialized neural network. Embodiments of the present disclosure provide an alternative augmentation strategy that does not involve a separate network training step, which increases the diversity in the training data samples, thereby helping in cases where there is a potential inherent confusion among a pair of classes.

[0074] For example, the data acquired by different sensors for a pair of confusing classes in our data set. The rows correspond to the two classes (top movement and bottom movement), and the columns correspond to the sensor types (acc: accelerometer, gyro: gyroscope, and la: linear acceleration). While it may be possible for a human to distinguish few movements intuitively, when put together in a sensor fusion setup, the collective multivariate signals may be hard to distinguish for the model. This becomes prominent because a sequence model requires an input window consisting of a fixed sequence length.D24040W001

[0075] While a very small sequence length would fail to provide enough context in our data, due to computational constraints, embodiments of the present disclosure also cannot extend the input sequence length infinitely. The desirable sequence length is determined empirically in the predistillation step, and then fixed afterwards in the later stages, and at the end device where the model would be deployed. Embodiments of the present disclosure also need to take care of the fact that at the end user, embodiments of the present disclosure may often encounter the scenario where the user moves the phone swiftly only for a small duration. In those cases, depending on the final android platform-based sampling rate, embodiments of the present disclosure could end up with only few discrete time steps. To address these challenging scenarios, the model must be equipped with the capability of learning with reasonably smaller, fixed sequence length, even if there are confusing classes.

[0076] In the cases of confusing classes, when embodiments of the present disclosure train a model with the cross-entropy loss and a one-hot encoded ground truth vector, the model is forced to infer that there is a certain class, even though it may not be truly aware of the data category due to the confusing nature. Training in this manner produces an overconfident model that eventually leads to suboptimal inference performance, and a lower accuracy. To address this, embodiments of the present disclosure seek to increase the data diversity and suggest using the Mix-Up augmentation technique.

[0077] For a pair of examples X; and Xj respectively having one-hot encoded class labels y and y, Mix-Up tries to mix the pair both in terms of the data, as well as the labels, resulting in a new example x with an associated mixed label y as follows:x = A Xj + (1 - A) Xj (12)y = A + (1 — A) y (13)

[0078] Here, A G [0,1] is sampled from a Beta distribution. Mixing samples and labels in this manner helps increase the data diversity and helps the model not to be overconfident in predicting an example. This teaches the model to understand the percentage of belongingness of an example towards a certain category. This augmentation technique despite being simple to implement, is very effective in boosting the predictive performance of a model, especially when done carefully to choose the pair of classes to target. Experimental results demonstrate significant performance improvements when Mix-Up is used for the confusing pair of classes in our data.Probability FilteringD24040W001

[0079] According to some aspects, given a pair of (for example, student, or any generic) model checkpoints, in cases of having confusing, difficult or underrepresented classes, and by virtue of the training strategy, embodiments of the present disclosure may have one checkpoint performing well across most classes, except for few others, and the other checkpoint performs better in the remaining ones where the first is not performing well. To leverage the strengths of both the models, for each class, embodiments of the present disclosure only filter out the better performing variant to contribute to the final value for the corresponding component of the distribution. This is done with the help of a hold-out validation set, which is assumed to be diverse, and possibly different from that of the training data, to ensure generalizability towards unseen test samples.

[0080] This idea can be implemented using a membership array Jl [1: C] of length equal to the total number of classes C. Each element of this array 풜(c) ∈ {1,2}, indicates the student variant embodiments of the present disclosure are allowing to contribute to the probability of the cth class in the final distribution.can be populated by inspecting the performances on a hold-out validation set, based on final decisions using different permutations of 풜. Thus, using our method, the cthcomponent of the final distribution, representing the predicted probability of the cth class for x,, can be computed as follows:= 핀[풜(c) = 1] * pi(1)(c) + 핀[풜(c) = 2] * pi(2)(c) (14)

[0081] Here, H[. ] is an indicator function that takes a value of 1 if the condition inside [. ] is met, and otherwise takes a value of 0. This ensures that cth component of the final distribution is determined by the better performing student variant, and thus, in a way performs filtering of the appropriate student. It should be noted that an additional normalizing of the terms of pi(final)can be done to make them adhere to the definition of mathematical probability, or used as is for decision making (as embodiments of the present disclosure are anyway taking an argmax just after that).

[0082] FIG. 5 illustrates a checkpoint ensembling process 500 according to embodiments of the present disclosure. In the checkpoint ensembling process 500, a test window containing sensor measurements is processed in parallel by first student model 235 (si) and second student model 240 (S2) to generate their respective class probability distributions pi(l)(c) and pt(2)(c). The respective class probability distributions pi(l)(c) and pi(2)(c) then flow into a fusion strategy module that offers three distinct approaches for combining the predictions: an average-based fusion that computes the mean of both models' probabilities, a maximum -based fusion that selects the higher probability value for each class, and the proposed probability filtering strategy that uses a class-D24040W001specific indicator function to selectively choose between the student models based on their validated performance for each activity class. This ensembling framework enables the system to effectively leverage the complementary strengths of both student models while mitigating their individual weaknesses in recognizing specific activity classes.

[0083] The checkpoint ensembling process 500 includes multiple fusion approaches for combining predictions from the first student model 235 and the second student model 240 (also shown in FIG. 2) when processing a test window 520. Each fusion strategy aims to generate a final probability distribution for activity classification. The average-based fusion 505 computes the final probability for each class by taking the arithmetic mean of the corresponding class probabilities from both student models. This approach produces a balanced combination of both models' predictions, represented as mean(pi(l)(c), pi(2)(c)) for each class c.

[0084] The maximum -based fusion 510 selects the higher probability value between the two student models for each class, represented as max(pi(l)(c), pi(2)(c)). This strategy emphasizes stronger predictions from either model.

[0085] The fusion strategy 515 implements a novel probability filtering approach that selectively chooses between student model predictions based on validation performance. For each class c, an indicator function l(A(c)) determines which student model's prediction to use, producing the final probability as l(A(c)=l) * pi(l)(c) + l(A(c)=2) * pi(2)(c). This approach enables the system to leverage the strengths of each student model for specific activity classes where they demonstrate superior performance.

[0086] The following describes an example operation for training a parallel student model in the same way but focus more on the challenging classes during the distillation via augmentation. With this, both students can compensate the performance of each other, jointly. For training the second student with augmentation, embodiments of the present disclosure specifically study synthetic time-series data generation using a generative model and the Mix-Up technique.

[0087] Having obtained the two student model checkpoints, each with their own strengths (in terms of being good at predicting certain classes), embodiments of the present disclosure now discuss a next step of our proposed framework, which is referred to as “filtering.” The focus of this step is to have an effective checkpoint ensemble mechanism to fuse the decisions made by the two students. The first student model checkpoint is trained with LCEand ℒdistilas s1(.), and the second student model checkpoint is trained with £CE, ℒdistiland ℒaugas s2(.).The total loss for sx(. ) is computed as:ℒtotal= αℒCE+ (1 − α)ℒcddistilD24040W001The total loss for s2(. ) is computed as:^ℒtotal= α(ℒCE+ λaugℒaug) + (1 − α)ℒcddistilHere, λaug> 0, is a hyperparameter for the loss term ℒaugconsisting of the augmented examples, computed as follows:ℒaug= − (1 / |ℬ|) ∑|B|i=1∑Cj=1^aug =i=l j=lℒaugand ℒCEare computed in the similar manner, with the exception that the former is computed using the real examples, and the latter using the augmented examples.

[0088] Accordingly,∈ ℝCis employed to denote the output probability distribution for the example xias obtained by the model sm(.), where m G {1,2}. Using sm(. ), embodiments of the present disclosure can then compute the probability of an example Xj being predicted as belonging to the cthclass using p^m\c\ the cthcomponent of vector pmwhich can be expressed via the soft-max function as:pi(m)(c) = σ(sm(xi); τ)(c) = exp(sm(xi)(c) / τ)pi(m)(c) = σ(sm(xi); τ)(c) =∑Ck=1exp(sm(xi)(k) / τ)

[0089] Let, pi(final)∈ ℝCdenote the final probability distribution for an example xiobtained by fusing the decisions of s1(. ) and s2(. ). embodiments of the present disclosure can use the following two strategies to perform the decision fusion:

[0090] Average-based Fusion: In this technique, the final distribution can be obtained by averaging the distributions of the two model variants. The cthcomponent, representing the predicted probability of the cthclass for xbcan then be computed as follows:pi(final)(c) = mean(pi(1)(c), pi(2)(c))

[0091] Max-based Fusion: In this technique, the final distribution can be obtained by filtering out the element-wise maximum of the distributions of the two model variants. The cth component, representing the predicted probability of the cth class for Xj, can then be computed as follows:ppi(final)(c) = max(pi(1)(c), pi(2)(c))For each of the above fusion strategies, embodiments of the present disclosure can further use the sharpening technique to reduce the entropy of the label distribution. Sharpening is performed as:pi(sharp)(c) =∑Ck=1pi(m)(k)1 / TD24040W001

[0092] Here, T is a hyperparameter, and is analogous to “temperature” of the categorical distribution. While each of the above fusion strategies are reasonable mechanisms to perform decision fusion, embodiments of the present disclosure found them to be ineffective when it comes to combining model predictions from a pair of variants that may be performing well overall, but one of them not performing good in certain classes. This is because even if a student model performs well on a challenging class where its counterpart could not perform well, simply computing the average or taking out the maximum does not help combine the best of both the checkpoints. Instead, what embodiments of the present disclosure observed is that the values of the final distribution get “squeezed” in a range inbetween the values of the individual ones, and often the performance is drawn downwards because of the underperforming variant.

[0093] Algorithm 1 is an example algorithm that can be used in the Distillation-Augmentation-Filtering (DAF) framework.D24040W001Algorithm 1 DAF FrameworkRequire: Labeled dataset {Xj, yj of windows and activity class label.Output: A pair of student models sx(. ) and s2(. ) and membership array < A[1: C], Steps:1: For c in 1: C, identify Mc(. ), a DL model architecture performing best for class c.2: Randomly initialize weights of s^.) and s2(.).3: / * Contrastive Distillation (CD) * / 4: for epoch in 1 to epochs si do5: for S in set of mini-batches do6: Obtain {Xj.yj G B7: Compute £total= a£CE+ (1 - a^dttii using {Xj, yj8: Update s^.) using V£totai9: Save model checkpoint for ) using validation performance.10: end for11: Identify set of classes { c } where s1(. ) does not perform satisfactorily.12: / * Contrastive Distillation with Augmentation (CDA) via Generative Model * / 13: For each c, synthesize {V(Xj),yj = c} using generative model V and obtain X£= {V(Xj), yj = c } U {Xj, yj = c}. X£contains real and synthetic data for class c.14: Train s2(.) using steps 4-10 with {Xg V c] U {Xj,yj = c $ { c } }. Call it s2* (.).15: / * Contrastive Distillation with Augmentation (CDA) via Mix-Up * / 16: Using { c }, form pairs of classes {(cl, c2)}to obtain mix-up dataset { x, y}.17: Use original {x,, yj, mix-up dataset { x,y}, repeat 4-10 to train s2(.). However, replace 7 with, £tot:ai^(^CE "b ^-aug -^aug ) d” (1 distil- Call it s2** (.). 18: Using heuristics from validation data, assign better performing of s2* (.) and s2** (■) as 52(.).19: I* Filtering (F) * / 20: all combinations = itertools.product([l, 2], repeat=C)D24040W00121: Find JI G all combinations that maximizes validation performance using s1(. ) and 52G).22: return s1(.), s2(. ) and JI.

[0094] Some embodiments of the present disclosure include speed-memory trade-offs associated with the described DAF framework. Regarding latency, embodiments of the present disclosure can perform inference by running both the students in parallel, to make sure that the latency remains the same as in the base LSTM. In some instances, due to the storage requirement of two student checkpoints, embodiments of the present disclosure do encounter a two-fold increased memory. However, as demonstrated in some experiment results, the DAF framework shows improvements in terms of having a better speed-memory trade-off. This may be because a naive doubling of units in the base LSTM not only increases memory, but also doubles the latency.Example WorkflowsD24040W001

[0095] FIG. 6 is a flowchart illustrating a human activity recognition method 600 of according to some examples. The human activity recognition method 600 may be implemented on a mobile device 100 equipped with sensors and an electronic processor.

[0096] At block 605, the human activity recognition method 600 includes receiving temporal sensor data within an interval of sensor measurements. In some examples, the temporal sensor data comprises multivariate time-series data from one or more sensors of the mobile device 100, such as accelerometers, gyroscopes, or cameras. In some examples, the interval of sensor measurements represents a fixed-duration time window containing raw sensor measurements that capture temporal patterns of user activities in a multi-dimensional format.

[0097] At block 610, the human activity recognition method 600 includes processing, with an electronic processor, the temporal sensor data through a first student model to obtain a first activity class probability. The first student model may comprise a neural network model derived through knowledge distillation from one or more teacher models. In some examples, the first student model is a lightweight neural network model trained to achieve similar performance to larger teacher models while requiring fewer computational resources during deployment.

[0098] At block 615, the human activity recognition method 600 includes processing, with the electronic processor, the temporal sensor data through a second student model to obtain a second activity class probability. In some examples, the second student model is trained using augmented data and also comprises a neural network model derived through knowledge distillation from one or more teacher models.

[0099] In some examples, the augmented data used to train the second student model is generated using a variational autoencoder that creates synthetic training data specifically for activity classes identified as having lower classification performance during validation.

[0100] At block 620, the human activity recognition method 600 includes performing, with the electronic processor, probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability. The probability filtering may include selecting, for each activity class, a probability from a student model based on pre-determined validation performance. Alternatively, the probability filtering may include averaging the first activity class probability and the second activity class probability, or selecting a maximum value between the first activity class probability and the second activity class probability. The class-specific selection indicator may be predetermined based onD24040W001relative performance of the first student model and the second student model on validation data for each activity class. In some examples, the class-specific selection indicator determines which student model's prediction to use based on each model's demonstrated performance for specific activity classes during validation, allowing the system to leverage the complementary strengths of both student models.

[0101] In some examples, performing the probability filtering comprises selecting, for each activity class, a probability from among the first activity class probability, the second activity class probability, and the at least one additional activity class probability based on validated performance of each respective student model for the activity class. This class-specific selection mechanism enables the system to dynamically choose the best-performing student model's prediction for each class, optimizing overall recognition performance while maintaining computational efficiency suitable for mobile deployment.

[0102] At block 625, the human activity recognition method 600 includes outputting, with the electronic processor, a recognized activity corresponding to the temporal sensor data based on the third activity class probability. In some examples, the recognized activity represents a classification of the user's current physical activity based on the processed sensor data, such as walking, running, sitting, or other activities supported by the trained models.

[0103] At block 630, the human activity recognition method 600 includes performing or initiating responsive actions based on the recognized activity. For example, the performing or initiating responsive actions includes generating a control signal in response to the recognized activity. The human activity recognition system provides rich interactive applications on mobile devices. Once an activity is recognized, the system may trigger various responsive actions. For example, the recognized activity can control the movement of 2D or 3D objects displayed on the mobile device screen, with different gestures corresponding to different object manipulations. In multimedia applications, the system can play specific audio through the device speakers based on the detected activity, creating an immersive user experience. Additionally, the appearance of virtual objects can be dynamically modified according to different recognized activities, enabling intuitive gesture-based interfaces for gaming, augmented reality experiences, and interactive content consumption. These responses create a natural bridge between physical human movements and digital interactions. In some examples, the performing or initiating responsive actions includes generating a control signal that cause a display device to display the recognized human activity. In some examples, the human activity recognitionD24040W001method 600 includes triggering a notification or alert when the recognized activity falls in a predetermined category.

[0104] In some examples, the human activity recognition method 600 further includes processing the temporal sensor data through at least one additional student model to obtain at least one additional activity class probability. This additional processing enables the method to leverage specialized expertise from more than two student models, each trained to excel at different aspects of activity recognition. The additional student models provide complementary probability distributions that can be selectively combined with those from the first and second student models to improve overall recognition accuracy.

[0105] FIG. 7 is a flowchart illustrating a training method 700 for training a human activity recognition system according to some examples. The training method 700 may be implemented on a training system with sufficient computational resources for training neural network models.

[0106] At block 705, the method includes training a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture. In some examples, the teacher models are large, complex neural network models trained to achieve high accuracy in activity recognition, though they may require substantial computational resources.

[0107] At block 710, the method includes identifying, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data. In some examples, the bestperforming teacher model is selected by evaluating each teacher model's classification accuracy on validation data for each specific activity class.

[0108] At block 715, the method includes training a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model.

[0109] In some examples, the knowledge distillation process employs contrastive learning to align similar teacher probability distributions and separate dissimilar teacher probability distributions, enabling effective transfer of knowledge from teacher to student models.

[0110] At block 720, the method includes training a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.

[0111] In some examples, the cross-entropy loss term is used compare the model's output with known correct labels encoded as one-hot vectors. The augmentation loss term helps the model learn from synthetically generated training samples that target challenging activity classes. The distillationD24040W001loss term enables efficient knowledge transfer by ensuring the student model's probability distributions match those of the more complex teacher models.

[0112] In some examples, the augmented training data is generated using a variational autoencoder (VAE) model specifically for activity classes identified as having lower classification performance during validation. The VAE model generates synthetic training data that maintains the statistical properties of the original sensor data.

[0113] In some examples, after training both student models, the method may include a post-training evaluation phase where class-specific selection indicators are determined based on each student model's performance on validation data, preparing the system for efficient inference deployment.

[0114] In some examples, the method includes training a second student model using first augmented data targeting a first set of challenging activity classes, and training at least one additional student model using second augmented data targeting a second set of challenging activity classes different from the first set. This specialization allows each student model to develop expertise in specific difficult cases, creating a complementary ensemble where each model's strengths can compensate for others' weaknesses.Example Hardware

[0115] FIG. 8 is a block diagram of an example computing device 800. The computing device 800 is an apparatus configured to perform at least some operations of the above-described methods, algorithms, and procedures according to some examples. The computing device 800 includes at least one processor and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus to perform at least some operations of the of human activity recognition method 600 and the training method 700 for training a human activity recognition system.

[0116] The computing device 800 of FIG. 8 is illustrated as having a number of components, but any one or more of these components may be omitted or duplicated, as suitable for the application and setting. In some embodiments, some or all of the components included in the computing device 800 may be attached to one or more motherboards and enclosed in a housing. In some embodiments, some of those components may be fabricated onto a single system-on-a-chip (SoC) (e.g., the SoC may include one or more processing devices 802 and one or more storage devices 804). Additionally, in various embodiments, the computing device 800 may not include one or more of the components illustrated in FIG. 8, but may include interface circuitry for coupling to the one or more components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-DefinitionD24040W001Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other appropriate interface). For example, the computing device 800 may not include a display device 810, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which an external display device 810 may be coupled.

[0117] The computing device 800 includes a processing device 802 (e.g., one or more processing devices). As used herein, the term “processing device” refers to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. In various embodiments, the processing device 802 may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), server processors, or any other suitable processing devices.

[0118] The computing device 800 also includes a storage device 804 (e.g., one or more storage devices). In various embodiments, the storage device 804 may include one or more memory devices, such as random-access memory (RAM) devices (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridging RAM (CBRAM) devices), hard drive-based memory devices, solid-state memory devices, networked drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device 804 may include memory that shares a die with the processing device 802. In such an embodiment, the memory may be used as cache memory and include embedded dynamic randomaccess memory (eDRAM) or spin transfer torque magnetic random-access memory (STT-MRAM), for example. In some embodiments, the storage device 804 may include non-transitoiy computer readable media having instructions thereon that, when executed by one or more processing devices (e.g., the processing device 802), cause the computing device 800 to perform any appropriate ones of the methods disclosed herein or portions of such methods.

[0119] The computing device 800 further includes an interface device 806 (e.g., one or more interface devices 806). In various embodiments, the interface device 806 may include one or more communication chips, connectors, and / or other hardware and software to govern communications between the computing device 800 and other computing devices. For example, the interface device 806 may include circuitry for managing wireless communications for the transfer of data to and from the computing device 800. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate dataD24040W001via modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not. Circuitry included in the interface device 806 for managing wireless communications may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards, Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). In some embodiments, circuitry included in the interface device 806 for managing wireless communications may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, circuitry included in the interface device 806 for managing wireless communications may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, circuitry included in the interface device 806 for managing wireless communications may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. In some embodiments, the interface device 806 may include one or more antennas (e.g., one or more antenna arrays) configured to receive and / or transmit wireless signals.

[0120] In some embodiments, the interface device 806 may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communication protocols. For example, the interface device 806 may include circuitry to support communications in accordance with Ethernet technologies. In some embodiments, the interface device (806) may support both wireless and wired communication, and / or may support multiple wired communication protocols and / or multiple wireless communication protocols. For example, a first set of circuitry of the interface device 806 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second set of circuitry of the interface device 806 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some other embodiments, a first set of circuitry of the interface device 806 mayD24040W001be dedicated to wireless communications, and a second set of circuitry of the interface device 806 may be dedicated to wired communications.

[0121] The computing device 800 also includes battery / power circuitry 808. In various embodiments, the battery / power circuitry 808 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 800 to an energy source separate from the computing device 800 (e.g., to AC line power).

[0122] The computing device 800 also includes a display device 810 (e.g., one or multiple individual display devices). In various embodiments, the display device 810 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0123] The computing device 800 also includes additional input / output (I / O) devices 812. In various embodiments, the I / O devices 812 may include one or more data / signal transfer interfaces, audio VO devices (e.g., microphones or microphone arrays, speakers, headsets, earbuds, alarms, etc.), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, etc.), image capture devices (e.g., one or more cameras), human interface devices (e.g., keyboards, cursor control devices, such as a mouse, a stylus, a trackball, or a touchpad), etc.

[0124] Depending on the specific embodiment, various components of the interface devices 806 and / or I / O devices 812 can be configured to output suitable signals, receive suitable signals, and receive and output data streams. In some examples, the interface devices 806 and / or VO devices 812 include one or more analog-to-digital converters (ADCs) for transforming received analog signals into a digital form suitable for operations performed by the processing device 802 and / or the storage device 804. In some additional examples, the interface devices 806 and / or I / O devices 812 include one or more digital-to-analog converters (DACs) for transforming digital signals provided by the processing device 802 and / or the storage device 804 into an analog form suitable for being communicated over the corresponding communication channels.

[0125] A person skilled in the art realizes that the present disclosure by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible and considered within the scope of the appended claims. Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims, and which may represent systems, methods, and devices, all arranged in accordance with aspects of the present disclosure.D24040W001

[0126] EEE1. A method for human activity recognition comprises receiving temporal sensor data within an interval of sensor measurements; processing, with an electronic processor, the temporal sensor data through a first student model to obtain a first activity class probability; processing, with the electronic processor, the temporal sensor data through a second student model to obtain a second activity class probability; performing, with the electronic processor, probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability; and outputting, with the electronic processor, a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

[0127] EEE2. The method according to EEE1, wherein performing the probability filtering comprises selecting, for each activity class, a probability from a student model based on a predetermined validation performance.

[0128] EEE3. The method according to any of EEE1 to EEE2, wherein performing the probability filtering comprises performing at least one of averaging the first activity class probability and the second activity class probability; and selecting a maximum value between the first activity class probability and the second activity class probability.

[0129] EEE4. The method according to any of EEE1 to EEE3, wherein the method is implemented on a mobile device that includes the electronic processor, and wherein the temporal sensor data comprise multivariate time-series data from one or more sensors of the mobile device.

[0130] EEE5. The method according to any of EEE1 to EEE4, wherein each of the first student model and the second student model comprises a respective neural network model derived through a respective knowledge distillation from one or more respective teacher models.

[0131] EEE6. The method according to any of EEE1 to EEE5, wherein the class-specific selection indicator is predetermined based on relative performance of the first student model and the second student model on validation data for each activity class.

[0132] EEE7. The method according to any of EEE1 to EEE6, further comprising: processing, with the electronic processor, the temporal sensor data through at least one additional student model to obtain at least one additional activity class probability; wherein the second student model is trained using first augmented data targeting a first set of challenging activity classes; wherein the at least one additional student model is trained using second augmented data targeting a second set of challenging activity classes different from the first set; and wherein performing the probability filtering comprises selecting, for each activity class, a probability from among the first activity class probability, theD24040W001second activity class probability, and the at least one additional activity class probability based on validated performance of each respective student model for the activity class.

[0133] EEE8. A method for training a human activity recognition system comprises training a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture; identifying, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data; training a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model; and training a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.

[0134] EEE9. The method according to EEE8, further comprising: training at least one additional student model using second augmented training data and a third loss function comprising a second cross-entropy loss term, a second augmentation loss term, and a second distillation loss term; wherein the augmented training data used for training the second student model targets a first set of challenging activity classes; and wherein the second augmented training data used for training the at least one additional student model targets a second set of challenging activity classes different from the first set.

[0135] EEE10. The method according to any of EEE8 to EEE9, wherein the generative machine learning model includes a variational autoencoder (VAE) model.

[0136] EEE11. The method according to any of EEE8 to EEE10, wherein the cross-entropy loss term substantially minimizes a difference between a student model prediction and a ground-truth activity class label, the augmentation loss term substantially minimizes a difference between a student model prediction and an augmented training sample for the activity class, and the distillation loss term substantially minimizes a difference between a student model prediction and a teacher model probability distribution.

[0137] EEE12. The method according to any of EEE8 to EEE11, wherein training the first student model and training the second student model each comprise using contrastive learning to align similar teacher probability distributions and separate dissimilar teacher probability distributions.

[0138] EEE13. The method according to any of EEE8 to EEE 12, further comprising determining a class-specific selection indicator for probability filtering based on relative performance of the first student model and the second student model on validation data, wherein determining the class-specific selection indicator comprises evaluating performance of each student model on validation data forD24040W001each activity class, and selecting, for each activity class, the student model having better validation performance.

[0139] EEE14. An apparatus for human activity recognition comprises at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to receive temporal sensor data within an interval of sensor measurements; process the temporal sensor data through a first student model to obtain a first activity class probability; process the temporal sensor data through a second student model to obtain a second activity class probability; perform probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability; and output a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

[0140] EEE15. The apparatus according to EEE14, wherein the performing the probability filtering comprises selecting, for each activity class, a probability from a student model based on a predetermined validation performance.

[0141] EEE16. The apparatus according to any of EEE14 to EEE15, wherein the performing the probability filtering comprises performing at least one of averaging the first activity class probability and the second activity class probability; and selecting a maximum value between the first activity class probability and the second activity class probability.

[0142] EEE17. The apparatus according to any of EEE14 to EEE16, wherein the temporal sensor data comprise multivariate time-series data from one or more sensors a mobile device.

[0143] EEE18. The apparatus according to any of EEE14 to EEE17, wherein each of the first student model and the second student model comprises a respective neural network model derived through a respective knowledge distillation from one or more respective teacher models.

[0144] EEE19. The apparatus according to any of EEE 14 to EEE18, wherein the class-specific selection indicator is predetermined based on relative performance of the first student model and the second student model on validation data for each activity class.

[0145] EEE20. The apparatus according to any of EEE 14 to EEE 19, wherein the at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus at least to: process, with the electronic processor, the temporal sensor data through at least one additional student model to obtain at least one additional activity class probability; wherein the second student model is trained using first augmented data targeting a first set of challenging activity classes; wherein the at least one additional student model is trained using second augmented data targeting a secondD24040W001set of challenging activity classes different from the first set; and wherein performing the probability filtering comprises selecting, for each activity class, a probability from among the first activity class probability, the second activity class probability, and the at least one additional activity class probability based on validated performance of each respective student model for the activity class.

[0146] EEE21. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any one of EEE 1 to EEE13.

[0147] EEE22. An apparatus for human activity recognition, the apparatus comprising at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to train a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture; identify, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data; train a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model; and train a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.

[0148] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.

[0149] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systemsD24040W001and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0150] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0151] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0152] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims.

[0153] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0154] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented disclosure(s). Some embodiments can also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium including being loaded into and / or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as aD24040W001computer or a processor, the machine becomes an apparatus for practicing the patented disclosure(s). When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0155] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range.

[0156] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0157] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.

[0158] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.”

[0159] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner.

[0160] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if’ may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].”

[0161] Also, for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energyD24040W001is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.

[0162] As used herein in reference to an element and a standard, the term compatible means that the element communicates with other elements in a manner wholly or partially specified by the standard and would be recognized by other elements as sufficiently capable of communicating with the other elements in the manner specified by the standard. The compatible element does not need to operate internally in a manner specified by the standard.

[0163] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and / or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0164] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessors) or a portion of a microprocessor(s), that requires software (e g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in thisD24040W001application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0165] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0166] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in “DETAILED DESCRIPTION” and / or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

D24040W001CLAIMS1. A method for human activity recognition, comprising:receiving temporal sensor data within an interval of sensor measurements;processing, with an electronic processor, the temporal sensor data through a first student model to obtain a first activity class probability;processing, with the electronic processor, the temporal sensor data through a second student model to obtain a second activity class probability;performing, with the electronic processor, probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability, wherein the class-specific selection indicator controls how the first and second activity class probabilities are combined for a given activity class; and outputting, with the electronic processor, a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

2. The method of claim 1, wherein performing the probability filtering comprises:selecting, for each activity class, a probability from a student model based on a predetermined validation performance.

3. The method of claim 1 or 2, wherein performing the probability filtering comprises performing at least one of:averaging the first activity class probability and the second activity class probability; and selecting a maximum value between the first activity class probability and the second activity class probability.

4. The method of any preceding claim,wherein the method is implemented on a mobile device that includes the electronic processor, andwherein the temporal sensor data comprise multivariate time-series data from one or more sensors of the mobile device.D24040W0015. The method of any preceding claim, wherein each of the first student model and the second student model comprises a respective neural network model derived through a respective knowledge distillation from one or more respective teacher models.

6. The method of any preceding claim, wherein the class-specific selection indicator is predetermined based on relative performance of the first student model and the second student model on validation data for each activity class.

7. The method of any preceding claim, further comprising:processing, with the electronic processor, the temporal sensor data through at least one additional student model to obtain at least one additional activity class probability;wherein the second student model is trained using first augmented data targeting a first set of challenging activity classes;wherein the at least one additional student model is trained using second augmented data targeting a second set of challenging activity classes different from the first set; andwherein performing the probability filtering comprises selecting, for each activity class, a probability from among the first activity class probability, the second activity class probability, and the at least one additional activity class probability based on validated performance of each respective student model for the activity class.

8. A method for training a human activity recognition system, the method comprising:training a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture;identifying, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data;training a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model; andtraining a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.

9. The method of claim 8, further comprising:D24040WO01training at least one additional student model using second augmented training data and a third loss function comprising a second cross-entropy loss term, a second augmentation loss term, and a second distillation loss term;wherein the augmented training data used for training the second student model targets a first set of challenging activity classes; andwherein the second augmented training data used for training the at least one additional student model targets a second set of challenging activity classes different from the first set.

10. The method of claim 8 or 9, wherein the augmented training data are generated using a generative machine learning model for a challenging activity class, the challenging activity class having lower classification performance during validation, wherein the generative machine learning model includes a variational autoencoder (VAE) model.

11. The method of any one of claims 8 to 10, wherein:the cross-entropy loss term substantially minimizes a difference between a student model prediction and a ground-truth activity class label,the augmentation loss term substantially minimizes a difference between a student model prediction and an augmented training sample for the activity class, andthe distillation loss term substantially minimizes a difference between a student model prediction and a teacher model probability distribution.

12. The method of any one of claims 8 to 11, wherein training the first student model and training the second student model each comprise using contrastive learning to align similar teacher probability distributions and separate dissimilar teacher probability distributions.

13. The method of any one of claims 8 to 12, further comprising:determining a class-specific selection indicator for probability filtering based on relative performance of the first student model and the second student model on validation data, wherein determining the class-specific selection indicator comprises evaluating performance of each student model on validation data for each activity class, and selecting, for each activity class, the student model having better validation performance.D24040WO0114. An apparatus for human activity recognition, the apparatus comprising:at least one processor; andat least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to receive temporal sensor data within an interval of sensor measurements;process the temporal sensor data through a first student model to obtain a first activity class probability;process the temporal sensor data through a second student model to obtain a second activity class probability;perform probability filtering on the first activity class probability and the second activity class probability based on a class-specific selection indicator to generate a third activity class probability, wherein the class-specific selection indicator controls how the first and second activity class probabilities are combined for a given activity class; andoutput a recognized activity corresponding to the temporal sensor data based on the third activity class probability.

15. The apparatus of claim 14, wherein the performing the probability filtering comprises: selecting, for each activity class, a probability from a student model based on a predetermined validation performance.

16. The apparatus of claim 14 or 15, wherein the performing the probability filtering comprises performing at least one of:averaging the first activity class probability and the second activity class probability; and selecting a maximum value between the first activity class probability and the second activity class probability.

17. The apparatus of any one of claims 14 to 16, wherein the temporal sensor data comprise multivariate time-series data from one or more sensors a mobile device.

18. The apparatus of any one of claims 14 to 17, wherein each of the first student model and the second student model comprises a respective neural network model derived through a respective knowledge distillation from one or more respective teacher models.D24040WO0119. The apparatus of any one of claims 14 to 18, wherein the class-specific selection indicator is predetermined based on relative performance of the first student model and the second student model on validation data for each activity class.

20. The apparatus of any one of claims 14 to 19, wherein the at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus at least to:process, with the electronic processor, the temporal sensor data through at least one additional student model to obtain at least one additional activity class probability;wherein the second student model is trained using first augmented data targeting a first set of challenging activity classes;wherein the at least one additional student model is trained using second augmented data targeting a second set of challenging activity classes different from the first set; andwherein performing the probability filtering comprises selecting, for each activity class, a probability from among the first activity class probability, the second activity class probability, and the at least one additional activity class probability based on validated performance of each respective student model for the activity class.

21. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any one of claims 1-13.

22. An apparatus for human activity recognition, the apparatus comprising:at least one processor; andat least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: train a plurality of teacher models using temporal training sensor data, each of the plurality of teacher models having a different architecture;identify, for an activity class, a best-performing teacher model from the plurality of teacher models based on validation data;D24040WO01train a first student model through knowledge distillation using a first loss function comprising a cross-entropy loss term and a distillation loss term, wherein the knowledge distillation is based on the identified best-performing teacher model; andtrain a second student model using augmented training data and a second loss function comprising a cross-entropy loss term, an augmentation loss term, and a distillation loss term.

Images