Hierarchical Scene Graphs for Robot Anomaly Detection and Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing inventory management systems rely on human intervention for anomaly detection and correction, as current computer vision-based methods struggle with adaptability to layout changes, class imbalances, and require direct symbolic representations for planning, limiting automation.
Innovation Solution
A processor-implemented method using hierarchical scene graphs to generate interpretable sub-symbolic representations, enabling automated anomaly detection and correction through a robot equipped with a robotic arm and gripper, leveraging neural networks for perception and task planning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If computer vision-based methods with deep neural networks are used for anomaly detection, then detection precision is improved, but adaptability to layout changes and new classes deteriorates
Solution Approach 1:
The system dynamically adapts to layout changes by continuously learning from new data. The neural network model is updated periodically to incorporate new planograms and anomaly types, allowing the system to maintain high detection precision while adapting to changing retail environments without requiring complete retraining.
Solution Approach 2:
The system performs preliminary learning during offline training phases where it pre-processes and learns from historical planogram data and anomaly examples. This preliminary action enables the model to be pre-adapted to various layouts before deployment, improving both detection precision and adaptability when new layouts are introduced.
2Difficulty of detecting and measuring
If traditional vision-based methods with distributed representations are used, then detection capability is improved, but ease of operation for planning deteriorates due to continuous-valued vectors
Solution Approach 1:
The system introduces an intermediary module that translates continuous-valued distributed representations into discrete symbolic representations. This intermediary layer enables the planning system to work with interpretable symbols while the neural network continues to use continuous representations for detection, bridging the gap between detection capability and planning ease of operation.
Solution Approach 2:
The system segments the processing pipeline into distinct modules: a perception module that uses continuous representations for detection, and a planning module that uses discrete symbolic representations. This segmentation allows each module to operate in its optimal representation space, maintaining detection capability while improving planning operation ease.
3Extent of automation
If autonomous correction with mobile manipulator robot is implemented, then extent of automation is improved, but device complexity increases
Solution Approach 1:
The mobile manipulator robot is designed with universal components that can perform multiple functions: navigation to shelf locations, detection of anomalies using integrated sensors, and correction through manipulation. This multi-functionality reduces overall system complexity compared to having separate specialized systems for each task.
Solution Approach 2:
The system merges the detection and correction functions into a single integrated autonomous system. The mobile manipulator robot combines navigation, perception, and manipulation capabilities in one platform, reducing the complexity that would arise from coordinating multiple separate systems while maximizing automation extent.
4Measurement precision
If planogram-dependent methods are used for anomaly detection, then detection precision is improved, but adaptability to new classes and layout updates deteriorates
Solution Approach 1:
The system dynamically adapts to new product classes and layout updates by implementing continuous learning mechanisms. When new planograms or product types are introduced, the system automatically retrains and updates its neural network models, maintaining high detection precision while being highly adaptable to new classes.
Solution Approach 2:
The system changes its detection parameters and model configurations based on the specific retail context. By adjusting hyperparameters, feature extraction methods, and model architecture according to the particular layout and product classes being monitored, the system maintains precision across diverse scenarios while adapting to new classes.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Conventional task planners assume that the task-plans provided are executable, hence these are not task-aware. Present disclosure alleviates the downward refinability assumption, that is, planning can be decomposed separate symbolic and continuous planning steps by introducing bi-level planning, a plan which is a series of actions that the robot needs to take to achieve the goal task is curated. Firstly, abstract symbolic actions are converted to continuous vectors and used therein to enable interaction with an environment. Images of objects placed in the environment are captured and concepts are learnt from the captured images and attributes of objects are detected. A hierarchical scene graph is generated from the concepts and attributes wherein the graph includes interpretable sub-symbolic representations and from these interpretable symbolic representations are obtained for identifying goal task. Anomalies are detected from the scene graph and robotic actions are generated to correct the detected anomalies.