Audio Caption Correction via False Alarm and Miss Detection

The patent introduces an intermediary detection system that acts as a mediator between the captioning model and the final caption output. This intermediary component analyzes the relationship between audio tags and caption detections to identify false positives and false negatives, thereby improving caption accuracy without directly modifying the captioning model itself.

The system implements a feedback mechanism where the detection results (false alarms and misses) are used to generate a corrected caption. This feedback loop allows the system to continuously improve caption quality by learning from its own errors, comparing detected elements against ground truth audio tags, and adjusting subsequent caption generation accordingly.

The patent segments the caption evaluation process into distinct functional components: generating audio tags from audio data, generating candidate captions from audio data, detecting false positives by comparing tags and detections, and detecting false negatives by comparing tags and detections. This segmentation allows each component to be optimized independently and makes the overall system more manageable despite its increased complexity.

The system performs self-evaluation and self-correction by automatically comparing its own caption output against independently generated audio tags. This self-service capability allows the system to identify and correct its own errors without external intervention, justifying the added complexity through autonomous improvement.