Audio-to-Notation System with Dual ML Error Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for generating musical notation from audio signals face challenges such as inaccurate transcription, inefficiency in cleaning up MIDI information, lack of real-time editing capabilities, and inability to produce audio waveforms with notation, often resulting in difficult-to-read notations with timing and performance mistakes.
Innovation Solution
A system utilizing machine learning models to process audio signals, including a first ML model for pitch and duration recognition and a second ML model for error correction, enabling real-time transcription and editing of musical notation, with tailored models for specific instruments and simultaneous processing of spectrograms to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional audio recognition methods are used to generate musical notation, then the notation can be generated from audio input, but the notation becomes difficult to read due to timing and performance mistakes
Solution Approach 1:
The system segments the musical notation generation process into multiple independent modules: audio signal processing, pitch detection, timing correction, and notation generation. Each module handles specific aspects separately, allowing timing mistakes to be corrected without affecting pitch detection accuracy, thereby improving overall notation readability while maintaining detection precision
Solution Approach 2:
The system introduces an intermediary timing correction layer between pitch detection and notation generation. This intermediary process identifies and corrects timing errors in detected notes before they are rendered in the final notation, acting as a mediator that preserves accurate pitch information while eliminating timing-related readability issues
2Productivity
If simple quantizers are used to clean up MIDI information, then the processing is fast, but the cleaning efficiency is insufficient
Solution Approach 1:
The system replaces traditional mechanical quantizer algorithms with a machine learning-based approach. The ML model learns optimal timing corrections from training data and applies them to MIDI information, achieving both high cleaning quality and maintained processing speed by substituting rigid mechanical rules with adaptive intelligent processing
3Ease of manufacture
If audio is converted to MIDI file and then to musical notation in a standalone offline process, then the conversion can be completed, but the notation cannot be easily edited
Solution Approach 1:
The system implements a dynamic, multi-stage processing pipeline where each stage (audio to MIDI, MIDI to notation) can be independently interrupted, modified, and reprocessed. This dynamic architecture allows users to edit intermediate representations at any point in the workflow, making the originally static offline process into an editable, flexible system
4Reliability
If conventional systems process audio signals through multiple separate steps, then comprehensive processing is achieved, but real-time processing and near-instantaneous conversion are not possible
Solution Approach 1:
The system merges multiple separate processing steps (audio analysis, pitch detection, timing correction, notation generation) into a single integrated real-time processing pipeline. By combining these functions into one unified system that processes signals continuously, the patent achieves both comprehensive transcription accuracy and near-instantaneous conversion capability
Data Source
Figure 1~3
Figure 4
AI summary
A system (100; 200) for generation of a musical notation from an audio signal, the system comprising at least one processor configured to: obtain the audio signal from an audio source (102) or a data repository (104); process the audio signal using first machine learning (ML) model(s) to generate a recognition result, wherein the recognition result is indicative of a pitch and a duration of a plurality of notes in the audio signal and their corresponding confidence scores; generate a preliminary musical notation using the recognition result; process the preliminary musical notation using second ML model(s) to determine whether the preliminary musical notation includes one or more errors; and when it is determined that the preliminary musical notation includes one or more errors, modify the preliminary musical notation to generate the musical notation that is error-free or has lesser errors as compared to the preliminary musical notation.