Monocular Aircraft Keypoint Detection for Automated Air Refueling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing aerial refueling systems rely heavily on human operators, which are costly and require additional equipment like stereoscopic vision or LIDAR, increasing complexity and expense.
Innovation Solution
An automated system using computer vision and a single camera to detect aircraft keypoints, merge them into a set of merged keypoints, determine the position of the fuel receptacle and boom tip, and control the refueling boom autonomously or with operator assistance, utilizing deep learning algorithms for accurate detection under adverse conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human operators are used for aerial refueling, then operational expertise and decision-making capability are improved, but system cost and operational complexity increase
Solution Approach 1:
The system enables the refueling operation to perform itself through automated computer vision-based detection and control. The monopolar camera system automatically detects aircraft keypoints, determines fuel receptacle positions, and controls the boom engagement without requiring human operators to manually perform these tasks, thus achieving self-service operation.
Solution Approach 2:
The patent replaces the mechanical human operator system with an automated computer vision system. The monopolar camera combined with AI-based keypoint detection and position determination algorithms substitutes the human operator's visual detection and decision-making functions, eliminating the need for complex operator accommodation while maintaining operational capability.
2Measurement precision
If stereoscopic vision with dual cameras is used, then depth perception and spatial awareness are improved, but system cost and component complexity increase
Solution Approach 1:
The patent extracts and eliminates the unnecessary dual-camera stereoscopic vision component from the system. By using a single monopolar camera with advanced computer vision algorithms for keypoint detection, the system removes the redundant second camera while still achieving the required depth perception and spatial awareness through monocular vision techniques.
Solution Approach 2:
The system uses computer vision algorithms to create a virtual 3D representation of the aircraft and fuel receptacle positions from 2D monopolar camera images. The keypoint detection and position determination processes generate accurate spatial information copies that replicate the depth perception capability of stereoscopic vision without requiring actual dual-camera hardware.
3Measurement precision
If LIDAR or radar is used for range measurements, then measurement accuracy is improved, but system cost and component complexity increase
Solution Approach 1:
The patent replaces active sensing systems like LIDAR and radar with a passive optical system. The monopolar camera combined with computer vision-based keypoint detection and position determination algorithms provides range measurement capability without requiring active electromagnetic radiation emission, thus eliminating LIDAR and radar components while maintaining measurement accuracy.
Solution Approach 2:
The system creates virtual range measurement data through image processing and geometric calculations based on detected aircraft keypoints. The position determination of the fuel receptacle is derived by copying and processing visual information from the monopolar camera, generating accurate range data without physical LIDAR or radar sensors.
4Reliability
If multiple sensors are used for detection, then detection accuracy and reliability are improved, but computational time and resource allocation increase
Solution Approach 1:
The patent extracts and removes unnecessary sensors from the detection system. By relying solely on a monopolar camera with sophisticated computer vision algorithms for keypoint detection, the system eliminates multiple sensors while reducing computational overhead. The single-camera approach processes less data than multiple sensors would generate, decreasing computational time and resource requirements.
Data Source
AI summary
Aspects of the disclosure provide solutions for automated air-to-air refueling (A3R) and assisted air-to-air refueling. Examples include: receiving a video frame; generating, from the video frame, a plurality of images having differing decreasing resolutions; detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled; merging the sets of aircraft keypoints into a set of merged aircraft keypoints; based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and determining a position of a boom tip of an aerial refueling boom. Some examples include, based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle, and for some examples, the video frame is monocular (e.g., provided by a single camera).