Vision-Based Joint Action and Pose Motion Forecasting

The patent combines independent action forecasting and pose forecasting into a unified joint forecasting system. The encoder processes video frames to extract features that are shared between action and pose prediction tasks, while the decoder generates both action labels and pose predictions simultaneously. This merging allows the system to leverage complementary information from both tasks, improving overall forecasting accuracy compared to independent systems.

The encoder-decoder architecture serves multiple functions: it extracts temporal features for action recognition, predicts future poses, and generates action forecasts. The same network components are utilized across different forecasting tasks, making the system multi-functional and efficient while maintaining high accuracy in both action and pose prediction.

The system performs preliminary action and pose forecasting at intermediate time steps during video processing, rather than waiting until the entire clip is analyzed. The encoder processes frames sequentially and the decoder generates predictions at each time step, enabling early warning of potential actions like falls before the action is fully completed. This preliminary forecasting maintains accuracy while providing timely responses for injury prevention.

The system uses feedback from intermediate prediction results to continuously refine action and pose forecasts. The predicted actions and poses at each time step are fed back into the network to improve subsequent predictions, allowing the system to maintain high accuracy while providing real-time predictions rather than delayed post-analysis results.