NFA-Based Action Detection and Counting for Low-Power Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action recognition and temporal action detection approaches require high computational load and large amounts of data, making them impractical for deployment on resource-constrained devices, and suffer from ambiguous labeling in sliding-window frame-wise predictions.
Innovation Solution
A machine learning scheme that classifies frames as corresponding to the start or end of sub-actions using a nondeterministic finite automaton (NFA) to simplify the learning task and reduce ambiguity, allowing for real-time action detection on low-powered devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If complex trained models and large amounts of data are used for action recognition, then generalization to different situations is improved, but computational load increases making deployment on resource-constrained devices impractical
Solution Approach 1:
The patent segments the action recognition task into two distinct stages: (1) a lightweight on-device detector that identifies candidate action regions in video frames, and (2) a more complex server-side model that performs detailed action classification and temporal detection on those candidates. This segmentation allows the resource-constrained device to handle only the lightweight detection stage while offloading computationally intensive classification to the server, thus resolving the contradiction between adaptability and device complexity.
2Speed
If sliding-window approach is used for frame-wise action prediction, then real-time detection is enabled, but ambiguous labeling occurs when windows contain mixed action and non-action frames
Solution Approach 1:
Instead of using the conventional sliding-window approach that predicts action labels for entire video windows (which causes ambiguity when windows contain mixed content), the patent inverts the approach by first detecting candidate action regions at the frame level using a lightweight detector, then applying temporal reasoning to identify complete actions. This inversion from window-based prediction to frame-based candidate detection eliminates the labeling ambiguity problem while maintaining real-time detection capability.
3Adaptability or versatility
If entire video windows are labeled for action detection, then comprehensive action coverage is achieved, but ambiguous labels reduce training effectiveness when windows contain both action and non-action frames
Solution Approach 1:
The patent applies local quality by focusing training on precisely defined action instances rather than entire video windows. The system identifies specific start and end frames of actions with high precision, then uses these localized action instances as training labels. This approach ensures that training data contains only clear action examples without ambiguous mixed-content windows, thereby improving training accuracy while maintaining comprehensive action coverage through the candidate detection and temporal reasoning process.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An electronic device is disclosed, comprising memory circuitry, and processor circuitry comprising an action detection circuitry configured to operate according to an action detection model for detecting an action based on a machine-learning scheme. The processor circuitry being configured to obtain sensor data; generate, based on the sensor data, a set of features associated with a frame; determine, based on the set, using the action detection model, whether the frame corresponds to a sub-action; apply a nondeterministic finite automaton, NFA, scheme, to the determined sub-action for the frame, wherein the NFA scheme has a set of states associated with corresponding sub-actions and is configured to output one or more action classes; determine, using the NFA scheme, an action class; detect the action based on the action class; and increment an action counter based on the detected action.