Multi-Modal Sound Effect Retrieval Beyond Text Metadata

The patent introduces an audio encoder as an intermediary component that transforms audio data into embeddings in a multi-modal embedding space. This encoder acts as a mediator between the audio content and the retrieval system, enabling direct audio-based search without relying on text metadata. The encoder projects audio into the same embedding space as visual and text data, allowing for accurate sound effect retrieval based on actual audio content rather than potentially inaccurate text descriptions.

The patent merges the audio encoder into the existing multi-modal embedding space that already contains visual and text encoders. By combining these encoding functions into a unified embedding space, the system achieves accurate audio-based retrieval without creating entirely separate complex systems. The shared embedding space allows different modalities (visual, text, audio) to be processed and retrieved together, reducing overall system complexity while maintaining high retrieval accuracy.

The patent implements a self-supervised learning pipeline that automatically generates high-quality audio-visual training pairs without requiring manual annotation. The system uses the multi-modal embedding space to identify and pair audio-visual matches autonomously, eliminating the need for time-consuming manual labeling while ensuring high data quality. This self-service approach significantly reduces training time and computational resources compared to supervised learning methods that require extensive manual data preparation.