Automated Air-to-Air Refueling Keypoint Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current boom refueling operations in aerial refueling rely heavily on human judgment and visual alignment, which can be challenging for boom operators situated aft of the cockpit, lacking a direct view of the boom and receiver.
Innovation Solution
The implementation of a machine learning-based Automated Air-to-Air Refueling (A3R) system that uses a camera to provide a video stream of the refueling boom and receiver, identifying keypoints on the receiver indicative of flight control surfaces, tracking their positions in real-time, and predicting changes in the receiver's 3D position to assist in precise alignment and control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If boom operators are situated aft of the cockpit to control the refueling boom, then they can operate the boom from a safe position, but they lack a direct view of the boom and receiver making visual alignment difficult
Solution Approach 1:
The patent introduces an automated vision system with cameras and image processing algorithms as an intermediary between the boom operators and the refueling target. The system captures images from multiple angles, processes them to identify key features, and provides augmented reality overlays that guide the operators without requiring them to have direct visual access to the target area.
Solution Approach 2:
The patent replaces the mechanical visual alignment process with an automated computer vision system. Instead of operators directly observing and manually aligning the boom with the receiver, the system uses image processing, feature detection, and automated calculation of alignment parameters to determine and communicate the correct positioning.
2Device complexity
If manual visual alignment is used by boom operators, then the system remains simple, but the accuracy of boom-to-receiver alignment is limited by human judgment
Solution Approach 1:
The patent replaces manual visual alignment with an automated computer vision system that uses image processing algorithms, feature detection, and computational geometry to calculate precise alignment parameters. The system processes images from multiple cameras, identifies key features on the receiver and boom, and computes the optimal positioning with higher precision than manual methods.
Solution Approach 2:
The patent creates digital representations (copies) of the physical refueling scene through multiple cameras and processes these digital copies to extract alignment information. The system generates virtual models of the boom and receiver positions based on image data, allowing for precise measurement and alignment calculations without physical contact or complex mechanical measurement devices.
3Loss of information
If real-time video streaming is provided to boom operators, then they have better visual information, but the operators are still out of direct view and must interpret the video feed
Solution Approach 1:
The patent introduces an automated vision system with cameras and image processing algorithms as an intermediary between the boom operators and the refueling target. The system captures images from multiple angles, processes them to identify key features, and provides augmented reality overlays that guide the operators without requiring them to have direct visual access to the target area.
Solution Approach 2:
The system provides real-time feedback to the operators through processed visual information and augmented reality overlays that indicate boom position, target location, and alignment status. This feedback loop allows operators to make informed control decisions based on automated analysis rather than raw video feeds alone.
Data Source
AI summary
An automated air-to-air refueling (A3R) system is usable with a tanker aircraft having a refueling boom. The system includes a camera connected to the tanker in proximity to the refueling boom which outputs a video stream of the boom and a fuel-receiving aircraft/receiver during an aerial refueling process. The system also includes a human-machine interface (“HMI”) located aboard the tanker, and an electronic control unit (“ECU”) in communication with the camera and HMI. The ECU identifies keypoints on the receiver indicative of flight control surfaces thereof, tracks corresponding positions of the flight control surfaces in real-time, and predicts a change in position of the receiver in free space as a predicted 3D position using the corresponding positions. The HMI also outputs a directional indicator indicative of the predicted 3D position, e.g., as a graphical overlay to a display screen.