Shearer Coal-Rock Interface Recognition Using Multi-Modal Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The recognition accuracy for a single image modality in identifying the coal rock interface is not high, leading to poor performance in automatically cutting the coal seam.
Innovation Solution
A method for training a coal rock interface recognition model using multi-modal data from a cloud server, including sample current, pressure, vibration, and video data, and performing decision-level fusion to generate a predicted coal rock distribution, which is then used for cutting control of a shearer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a pre-trained image recognition model is used to identify images from a single image modality, then the system is simple to operate, but the recognition accuracy for the coal rock interface is not high
Solution Approach 1:
The patent combines multiple data modalities (video data from cameras, vibration data from sensors, noise data from acoustic sensors, current data from motor sensors, and pressure data from hydraulic sensors) into a unified multi-modal data processing system. This merging of diverse data sources significantly improves coal rock interface recognition accuracy by providing complementary information that single modality systems cannot capture
Solution Approach 2:
The patent creates a multi-functional recognition system that processes various types of data (visual, vibrational, acoustic, electrical, and hydraulic) through a unified deep learning framework. This multi-functional approach allows the system to extract features from different physical domains and integrate them for comprehensive coal rock interface identification
2Measurement precision
If multi-modal data fusion is implemented to improve recognition accuracy, then the measurement precision increases, but the device complexity and data processing requirements increase
Solution Approach 1:
The patent segments the complex multi-modal data processing into distinct feature extraction modules for each data type (video feature extraction, vibration feature extraction, noise feature extraction, etc.). Each module independently processes its specific data type and extracts relevant features, which are then fused at the decision level. This segmentation reduces overall system complexity by making each processing stage more manageable and modular
Solution Approach 2:
The patent introduces decision-level fusion as an intermediary mechanism that bridges multiple feature extraction modules. Instead of directly combining raw multi-modal data which would be computationally intensive, the system first extracts features from each modality separately, then uses decision-level fusion to integrate these processed features. This intermediary approach significantly reduces data processing complexity while maintaining high recognition accuracy
3Speed
If real-time multi-modal data processing is performed at the edge processor, then the response speed increases, but the computational load and energy consumption increase
Solution Approach 1:
The patent implements partial processing at the edge processor by performing only feature extraction on multi-modal data locally, rather than complete model inference. The edge processor extracts relevant features from video, vibration, noise, current, and pressure data, then transmits these extracted features to a cloud server for final recognition and control decisions. This partial action approach enables fast local response while reducing energy consumption by avoiding full-scale computational processing at the edge
Data Source
AI summary
A method for training a coal rock interface recognition model is performed by a cloud server. The method includes receiving a sample coal rock distribution and sample multi-modal data from an edge processor, acquiring a sample load state feature based on the sample vibration data and the sample noise data; acquiring a sample cutting feature of a coal rock interface based on the sample video data. The method further includes acquiring a sample load feature for drum cutting based on the sample current data and the sample pressure data; calling the coal rock interface recognition model, and performing a decision-level fusion based on the sample load state feature; and training the coal rock interface recognition model based on the sample predicted coal rock distribution and the sample coal rock distribution to obtain a target coal rock interface recognition model.